A hybrid expert sparsification method for dense retrieval models

Abstract

The application provides a hybrid expert sparsification method for a dense retrieval model, which can be used in the technical field of large model inference acceleration and dense retrieval. The method comprises: performing clustering processing on each layer feedforward neural network of the dense retrieval model to obtain semantic experts; using the retention rate and the average activation frequency of each semantic expert to distinguish active and redundant semantic experts, and generating neuron masks of each layer feedforward network; performing single-head decoupling evaluation on each attention head of each layer multi-head attention, to obtain a retrieval experience score of each head; based on the score, carrying out beam search to generate a candidate subset, completing collaborative evaluation through inverse ranking fusion, and obtaining a comprehensive retrieval score of each subset; iteratively adding attention heads to the candidate subset and repeating the beam search and collaborative evaluation process until a preset condition is reached, screening an optimal subset and generating attention masks of each layer multi-head attention; and sparsifying the semantic experts and the multi-head attention layer by using the two types of masks, to obtain a sparsified dense retrieval model.

Description

Technical Field

[0001] This application relates to the fields of large model reasoning acceleration and dense retrieval technology, specifically to a hybrid expert sparsification method for dense retrieval models. Background Technology

[0002] With the development of pre-trained language models and large language models, semantic retrieval based on dense vector representation has become a crucial foundational module in search systems (e.g., e-commerce platforms, enterprise internal knowledge bases), question-answering systems (e.g., intelligent customer service systems, enterprise knowledge management systems, smart education systems, medical assistance systems, content creation systems), and retrieval augmented generation (RAG) systems. Existing dense retrieval methods typically encode the query text and candidate document text as continuous vectors, and calculate their semantic relevance using inner product or cosine similarity. Relevant dense retrieval models, through large-scale contrastive learning, instruction fine-tuning, and multi-task training, have achieved good retrieval performance on public retrieval benchmarks such as heterogeneous information retrieval benchmark frameworks.

[0003] However, existing model compression methods still have the following problems in accelerating dense retrieval models: existing pruning methods rely on shallow statistical indicators, which are not entirely consistent with the actual ranking indicators of the retrieval task; existing Mixture of Experts (MoE) or expertization methods often require additional training, and training-free schemes are difficult to guarantee the stability of the retrieval vector space; attention-based head pruning methods usually ignore multi-head collaboration relationships and lack a global selection mechanism based on retrieval results. Summary of the Invention

[0004] In view of the above problems, embodiments of this application provide a hybrid expert sparsification method for dense retrieval models.

[0005] The first aspect of this application provides a hybrid expert sparsity method for dense retrieval models, comprising: clustering each layer of the feedforward neural network in the dense retrieval model to obtain multiple semantic experts whose semantic boundaries do not overlap, wherein the dense retrieval model is used to encode the user-input query text and document text into dense semantic vectors and use the dense semantic vectors stored in memory to complete the user-specified document retrieval task; calculating the average activation frequency of each semantic expert, comparing the average activation frequency with a preset retention rate to obtain active semantic experts and redundant semantic experts, and using the active semantic experts and redundant semantic experts to generate a binary neuron mask for each layer of the feedforward neural network; and processing each attention of each multi-head attention layer in the dense retrieval model. The attention head performs a single-head decoupling evaluation operation to obtain the retrieval experience score of each attention head. Based on the retrieval experience score, a beam search is performed on each attention head to obtain multiple candidate subsets. A co-evaluation operation of inverse ranking fusion is then performed on the candidate subsets to obtain the comprehensive retrieval score of each candidate subset. The new attention head addition operation, beam search operation, and co-evaluation operation are iteratively performed on each candidate subset until a preset condition is met to obtain the optimal subset. The optimal subset is then used to generate a binary attention mask for each multi-head attention layer. The binary neuron mask is used to sparsify the semantic expert corresponding to each feedforward neural network layer, and the binary attention mask is used to sparsify each multi-head attention layer layer to obtain a sparsified dense retrieval model.

[0006] According to an embodiment of this application, the above-described clustering process of each layer of the feedforward neural network of the dense retrieval model to obtain multiple semantic experts whose semantic boundaries do not overlap includes: setting an objective function for optimizing the structured semantic boundaries; and, based on the semantic activation mode, using the objective function to cluster the column vectors of the upprojection matrix in each layer of the feedforward neural network on the training dataset to obtain multiple semantic experts whose semantic boundaries do not overlap.

[0007] According to an embodiment of this application, the above-mentioned method of calculating the average activation frequency of each semantic expert and comparing the average activation frequency with a preset retention rate to obtain active semantic experts and redundant semantic experts includes: traversing the model in a pure forward propagation manner, calculating the average activation frequency of each semantic expert on the task-related target training dataset, and obtaining the average activation frequency of each semantic expert; sorting each semantic expert in descending order according to the average activation frequency, and obtaining the descending order ranking result of the semantic experts; filtering the descending order ranking result of the semantic experts according to the preset retention rate to obtain active semantic experts, and defining other inactive semantic experts as redundant semantic experts.

[0008] According to an embodiment of this application, the above-mentioned statistical analysis of the average activation frequency of each semantic expert on the target training dataset related to the task includes: during the process of the dense retrieval model performing a gradient-free pure forward propagation operation on the current training sample in the target training dataset, obtaining the activation values ​​of all neurons in the current layer semantic expert after calculation by the activation function; using an indicator function to judge the activation values ​​to obtain the activation status of all neurons, and determining the activation proportion of the current layer semantic expert on the current training sample based on the activation status of all neurons; and processing the activation proportion of the current layer semantic expert on all training samples to obtain the average activation frequency of the current layer semantic expert on the target training dataset.

[0009] According to embodiments of this application, the above-described single-head decoupling evaluation operation performed on each attention head of each multi-head attention layer in the dense retrieval model to obtain the retrieval experience score of each attention head includes: isolating each attention head of each multi-head attention layer, retaining only one attention head as the data processing attention head during the forward propagation operation of the dense retrieval model; generating a query vector associated with the query text and a document vector associated with the document text using the data processing attention head, wherein the query text is obtained from the validation dataset and the document text is obtained from the document set; performing an inner product operation on the query vector and the document vector to obtain the semantic similarity between the query text and the document text; sorting the document set using the semantic similarity to obtain a document ranking list corresponding to each query text in the validation dataset; calculating the retrieval index of each query text using the document ranking list corresponding to each query text, and processing the retrieval indexes of all query texts to obtain the retrieval experience score of the data processing attention head.

[0010] According to an embodiment of this application, the above-mentioned beam search for each attention head based on retrieval experience score to obtain multiple candidate subsets includes: sorting the attention heads in descending order based on retrieval experience score to obtain a descending order sorting result of attention heads, and selecting multiple top-ranked attention heads from the descending order sorting result of attention heads according to a preset search beam width to obtain multiple candidate attention heads; and taking each candidate attention head as a candidate subset to obtain multiple candidate subsets.

[0011] According to embodiments of this application, the above-described collaborative evaluation operation of performing inverse ranking fusion on candidate subsets to obtain the comprehensive retrieval score for each candidate subset includes: generating a document ranking list corresponding to the query vector using each candidate attention head in the candidate subset, and obtaining the ranking value of each document text under each candidate attention head; calculating the contribution value of different candidate attention heads in the candidate subset to the ranking value for a single document text under the same query vector based on the set smoothing value; summing all contribution values ​​corresponding to a single document text to obtain the fusion score of a single document text; sorting all document texts in descending order based on the fusion score, and obtaining the comprehensive retrieval score of the candidate subset based on the result of the descending order sorting of document texts.

[0012] According to embodiments of this application, the above-described hybrid expert sparsification method for dense retrieval models further includes: generating binary neuron masks and binary attention masks associated with the target task and target dataset offline without updating the original parameters of the dense retrieval model and without performing gradient backpropagation; receiving new query text online and generating effective output for sparse vector retrieval using the binary neuron masks and binary attention masks through online inference.

[0013] According to embodiments of this application, the above-mentioned online reception of new query text and generation of effective output for sparse vector retrieval using binary neuron masks and binary attention masks in an online reasoning manner includes: determining the active semantic expert of each feedforward neural network layer in the dense retrieval model based on the binary neuron mask; processing the new query text using the active semantic expert to obtain a standard dense embedding vector; determining the effective attention head in each multi-head attention layer in the dense retrieval model using the binary attention mask; projecting the query vector, value vector, and key vector generated by processing the standard dense embedding vector with the effective attention head to obtain the projection result of the effective attention head; and fusing the projection results of all effective attention heads to obtain the effective output of each multi-head attention layer.

[0014] According to embodiments of this application, the dense retrieval models include dense retrieval models for biomedical retrieval, dense retrieval models for financial retrieval, dense retrieval models for fact-checking, dense retrieval models for entity retrieval, and dense retrieval models for long document retrieval.

[0015] A second aspect of this application provides a hybrid expert sparsity device for dense retrieval models, comprising: a semantic clustering module for clustering each layer of the feedforward neural network in the dense retrieval model to obtain multiple semantic experts whose semantic boundaries do not overlap, wherein the dense retrieval model is used to encode the query text and document text input by the user into dense semantic vectors and use the dense semantic vectors stored in memory to complete the task of recalling the user-specified document; a neuron mask generation module for calculating the average activation frequency of each semantic expert, comparing the average activation frequency with a preset retention rate to obtain active semantic experts and redundant semantic experts, and using the active semantic experts and redundant semantic experts to generate a binary neuron mask for each layer of the feedforward neural network; and a single-head decoupling evaluation module for evaluating each attention of each multi-head attention layer in the dense retrieval model. The system performs a single-head decoupling evaluation operation to obtain the retrieval experience score for each attention head. The inverse ranking fusion module performs a beam search on each attention head based on its retrieval experience score, obtaining multiple candidate subsets. It then performs a co-evaluation operation of inverse ranking fusion on these candidate subsets to obtain a comprehensive retrieval score for each subset. The attention mask generation module iteratively performs new attention head addition, beam search, and co-evaluation operations on each candidate subset until a preset condition is met, obtaining the optimal subset. The optimal subset is then used to generate a binary attention mask for each multi-head attention layer. Finally, the model sparsification module sparsifies the semantic experts corresponding to each feedforward neural network layer using binary neuron masks and sparsifies each multi-head attention layer using binary attention masks, resulting in a sparsified dense retrieval model.

[0016] According to a third aspect of this application, an electronic device is provided, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0017] According to a fourth aspect of this application, a computer-readable storage medium is also provided, on which a computer program or instructions are stored, wherein the computer program or instructions, when executed by a processor, implement the steps of the above-described method.

[0018] According to a fifth aspect of this application, a computer program product is also provided, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method.

[0019] The hybrid expert sparsity method for dense retrieval models provided in this application does not use internal statistics such as model weight amplitude and activation amplitude as selection criteria. Instead, it relies on the task activation frequency of semantic experts in the retrieval dataset to match the downstream retrieval needs of document recall. The entire process uses retrieval task metrics as the evaluation benchmark. First, a retrieval experience score based on recall metrics is obtained through single-head decoupling evaluation. Then, a reciprocal ranking fusion (RRF) method is used. RankFusion and the comprehensive retrieval score measure the effectiveness of the head combination. The selection criteria are fully aligned with the downstream retrieval targets of document recall, avoiding the accidental deletion of structural units that are crucial to retrieval ranking. This solves the problem of inconsistency between pruning criteria and downstream retrieval targets in existing technologies. At the same time, the hybrid expert sparsity method for dense retrieval models provided in this application does not require gradient backpropagation or model parameter update operations. The semantic experts of the feedforward neural network are obtained only by clustering the weight matrix of the original model. There is no need to fine-tune the router or retrain the expert allocation strategy. Activation frequency statistics, attention head evaluation, and bundle search optimization all only perform model forward propagation. Only a static binary mask is generated offline. The weights of the original dense retrieval model remain unchanged throughout the process. Structured sparsity is achieved by relying on the mask. Expert compression can be completed without additional training or fine-tuning. It can be directly adapted to the deployed encoding model, thus solving the problem of expert number compression relying on additional training in existing technologies. Furthermore, the hybrid expert sparsification method for dense retrieval models provided in this application abandons the approach of directly pruning based solely on single-head metrics. Instead, it first calculates the single-head retrieval score as an initial reference, without directly eliminating attention heads. It constructs a candidate subset of multiple attention heads, uses RRF to eliminate differences in scoring scales among different attention heads, quantifies the overall retrieval effect of the multi-head combination, it iteratively expands and evaluates the multi-head combination through bundle search, comprehensively considers the complementary and synergistic effects of multiple heads, and selects the optimal head combination based on the overall retrieval score of the subset. This approach takes into account the synergistic contributions among multiple attention heads, thereby solving the problem in existing technologies where attention head selection ignores the synergistic relationship among multiple heads. Attached Figure Description

[0020] The above-mentioned contents, other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0021] Figure 1 A diagram illustrating an application scenario of a hybrid expert sparsification method for dense retrieval models according to an embodiment of this application is shown.

[0022] Figure 2 A flowchart of a hybrid expert sparsification method for a dense retrieval model according to an embodiment of this application is shown;

[0023] Figure 3An architecture diagram of a sparse hybrid expert compression scheme for dense retrieval according to an embodiment of this application is shown;

[0024] Figure 4 A structural block diagram of a hybrid expert sparsification device for a dense retrieval model according to an embodiment of this application is shown; and

[0025] Figure 5 A block diagram of an electronic device suitable for implementing a hybrid expert sparsification method for dense retrieval models, according to an embodiment of this application, is shown. Detailed Implementation

[0026] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0027] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0028] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0029] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0030] As used herein, the term "model" refers to a model that learns the relationship between inputs and outputs from training data, thereby generating a corresponding output for a given input after training. Model generation can be based on machine learning techniques. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs using multiple layers of processing units. A neural network model is an example of a deep learning-based model. In this application, "model" may also be referred to as a "machine learning model," "learning model," "machine learning network," or "learning network," and these terms are used interchangeably throughout this application.

[0031] It's important to note that the term "neural network" can refer to a machine learning network based on deep learning. A neural network processes input and provides corresponding output, typically consisting of an input layer, an output layer, and one or more hidden layers between them. Neural networks used in deep learning applications often include many hidden layers, increasing the network's depth. The layers of a neural network are connected sequentially, so that the output of the previous layer serves as the input to the next layer. The input layer receives the input to the neural network, while the output layer's output becomes the final output. Each layer of a neural network includes one or more nodes (also called processing nodes or neurons), each processing the input from the layer above.

[0032] It should be understood that machine learning generally includes three phases: training, testing, and application (also known as inference). In the training phase, a given model is trained using a large amount of training data, iteratively updating parameter values ​​until the model can consistently generate inferences that meet the expected goals from the training data. Through training, the model can be considered to have learned the relationship between inputs and outputs (also known as the input-output mapping) from the training data. The parameter values ​​of the trained model are determined. In the testing phase, test inputs are applied to the trained model to test whether it can provide the correct output, thus determining the model's performance. In the application phase, the model can be used to process actual inputs based on the trained parameter values ​​to determine the corresponding output.

[0033] In one or more embodiments described in this application, the term "large model" can refer to a deep learning model with a large number of model parameters, which can include hundreds of millions, tens of billions, hundreds of billions, trillions, or even tens of trillions of model parameters. A large model can also be called a foundational model / basic model. It is pre-trained using large-scale unlabeled corpora to produce a pre-trained model with hundreds of millions of parameters. Such a model can adapt to a wide range of downstream tasks and has good generalization ability, such as large language models and multimodal pre-trained models. It should be understood that in practical applications, large models only require a small number of samples to fine-tune the pre-trained model before it can be applied to different tasks. Large models can be widely used in natural language processing, computer vision, and other fields. Specifically, they can be applied to computer vision tasks such as visual question answering, image description, and image generation, as well as natural language processing tasks such as text-based sentiment classification, text summarization generation, and machine translation. Major application scenarios for large models can include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0034] As the size of embedded model parameters continues to increase, the computational overhead and memory usage during online inference also increase significantly. Standard Transformer forward propagation typically computes the self-attention module and feedforward network module for each input, involving most model parameters in the computation. In applications such as RAG, online search, and large-scale document encoding, this full activation approach leads to high inference costs, limiting model deployment in low-latency, high-concurrency, or resource-constrained environments. To reduce model inference overhead, existing technologies include model pruning, hybrid experts, and attention head pruning. Model pruning methods typically remove some parameters based on weight magnitude, activation statistics, gradient sensitivity, or second-order approximation information; these methods can compress large-scale models with relatively low training costs. However, these methods are primarily geared towards general language modeling tasks, and their pruning is mostly based on internal model statistical signals, failing to directly align with ranking metrics important for dense retrieval tasks, such as Recall@K (a coverage metric in information retrieval and recommendation systems that measures whether the top K results contain the correct answer, where K is a positive integer) and NDCG@K (a scoring tool commonly used in recommendation systems and search engines to measure the accuracy and order of the top K results, where K is a positive integer). Therefore, at high sparsity rates, traditional pruning methods may disrupt the semantic boundary between query vectors and document vectors, leading to a decline in retrieval quality. Hybrid expert models, through gating routing mechanisms, activate only a subset of experts during each forward propagation, thereby reducing the computational cost per pass. Related models have demonstrated the effectiveness of sparse expert structures in large-scale models. Some methods also attempt to convert existing dense models into MoE models, for example, by replicating dense weights, partitioning experts, or obtaining multiple expert modules through further training. However, such methods typically require additional router training or fine-tuning, making them difficult to apply directly to deployed embedding models. Training-free expert segmentation methods, on the other hand, rely primarily on parameter clustering and lack feedback from downstream retrieval results, potentially affecting the discriminative power of the vector space. Furthermore, existing attention head pruning methods usually evaluate the importance of each attention head based on attention entropy, L1 norm, gradient importance, or single-head ablation results, and remove some attention heads accordingly. However, these methods often evaluate individual attention heads, neglecting the collaborative relationships between multiple attention heads. For dense retrieval tasks, different attention heads may capture semantic information of varying granularities, and attention heads with low individual scores may still contribute to the final retrieval ranking in specific combinations. Therefore, pruning based solely on single-head statistics is insufficient to obtain globally optimal head combinations for retrieval results.

[0035] Based on the problems existing in the current technology, there is an urgent need for a training sparsity-free method for dense embedding models. This method should be able to use downstream retrieval results as structure selection signals to perform task-related structured selection of feedforward network modules and attention heads without updating the original model parameters, thereby reducing inference computation overhead and maintaining retrieval performance as much as possible.

[0036] Figure 1 The diagram illustrates an application scenario of the hybrid expert sparsification method for dense retrieval models according to an embodiment of this application.

[0037] like Figure 1 As shown, application scenario 100 according to an embodiment of this application may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables. For example, a user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send information, etc.

[0038] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be electronic devices such as smartphones, wearable devices, personal computers, intelligent voice interaction devices, smart home appliances, intelligent vehicles, in-vehicle terminals, aircraft, unmanned vending terminals, and extended reality devices. Extended reality devices can include virtual reality devices, augmented reality devices, and mixed reality devices. A client application for the target application can be installed and run on the terminal devices. This target application can include, but is not limited to, financial transaction applications, payment applications, shopping applications, web browser applications, search applications, instant messaging tools, email clients, and social media platform software (these are just examples). Furthermore, this application embodiment does not limit the form of the target application, and it can include, but is not limited to, applications, mini-programs, etc., installed on the terminal devices, and can also be in the form of web pages.

[0039] Server 105 can be a server providing various services, such as a backend management server supporting websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process received user requests and other data, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services such as cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and basic cloud computing services such as big data. The server can be the backend server of the aforementioned target application, used to provide backend services to the clients of the target application.

[0040] It should be noted that the hybrid expert sparsification method for dense retrieval models provided in this application embodiment can generally be executed by server 105 and / or terminal devices 101-103. Accordingly, the hybrid expert sparsification device for dense retrieval models provided in this application embodiment can generally be set in server 105 and / or terminal devices 101-103.

[0041] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0042] Figure 2 A flowchart of a hybrid expert sparsification method for dense retrieval models according to an embodiment of this application is shown.

[0043] like Figure 2 As shown, the hybrid expert sparsification method 200 for dense retrieval models according to the embodiments of this application may include operations S210 to S260.

[0044] In operation S210, each layer of the feedforward neural network of the dense retrieval model is clustered to obtain multiple semantic experts whose semantic boundaries do not overlap.

[0045] The dense retrieval model encodes the user-input query text and document text into dense semantic vectors and uses the dense semantic vectors stored in memory to complete the task of recalling user-specified documents.

[0046] The aforementioned dense retrieval model is typically deployed on a high-performance graphics processing unit (GPU) cluster and interacts with the outside world or its own internal systems through high-performance storage (such as high-performance memory).

[0047] The dense retrieval models mentioned above are, for example, dense embedding models based on the Transformer architecture.

[0048] The aforementioned dense retrieval model can be a dense retrieval model specifically designed for search, such as a search system applied to e-commerce platforms or enterprise internal knowledge bases; it can also be a dense retrieval model specifically designed for question answering, such as a question answering system applied to intelligent customer service systems, enterprise knowledge management systems, smart education systems, medical assistance systems, or content creation systems; or it can be a dense retrieval system specifically designed for RAG (Research, Analysis, and Entry). Correspondingly, the user-input query text, document text, and user-specified document retrieval task will differ depending on the application scenario.

[0049] The aforementioned feed-forward neural network (FFN) is an example. All neural networks capable of feature extraction and feature projection can be clustered to divide them into different semantic experts with clear semantic boundaries.

[0050] The clustering process described above can be performed using, for example, the K-Means clustering algorithm.

[0051] Before performing operation S210, firstly, select a small amount of calibration data, documents and their related labeled samples from the target retrieval task; secondly, set the number of experts N in the feedforward neural network (N is a positive integer, such as 64 or 128); finally, set the expert retention rate and attention head retention rate of the feedforward network (such as 0.5, 0.7 or 0.9); and set the beamwidth of the attention head combination search.

[0052] For each layer of the dense retrieval model (or dense embedding model, dense embedding retrieval model), extract the upprojection matrix from the feedforward neural network of that layer. Treat each column vector in the upprojection matrix as the semantic projection direction of an intermediate neuron, and use the K-Means clustering method to divide the intermediate neurons of that layer into N non-overlapping semantic experts. Record the expert IDs and create a feedforward neural network expert index table.

[0053] In operation S220, the average activation frequency of each semantic expert is calculated, and the average activation frequency is compared with the preset retention rate to obtain active semantic experts and redundant semantic experts. The active semantic experts and redundant semantic experts are then used to generate the binary neuron mask for each layer of the feedforward neural network.

[0054] The calibration data is input into the original dense embedding model, and the average activation frequency of each semantic expert in the target task is calculated using gradient-free forward propagation. Then, the experts in the same layer are ranked according to their activation frequency, and the top-ranked experts are selected as active experts based on a preset expert retention rate. The neuron mask corresponding to the active expert is set to 1, while the neuron masks corresponding to the other experts are set to 0, thus generating the binary neuron mask for each layer of the feedforward neural network.

[0055] In operation S230, a single-head decoupling evaluation operation is performed on each attention head of each multi-head attention layer in the dense retrieval model to obtain the retrieval experience score for each attention head.

[0056] For each multi-head attention layer (or multi-head self-attention layer), a temporary attention head mask is used to evaluate the retrieval contribution of each attention head individually. On the calibration dataset, the output contribution of each individual attention head is retained, and the similarity between the query vector and the document vector is calculated to obtain the single-head retrieval score for each attention head.

[0057] In operation S240, based on the retrieval experience score, a beam search is performed on each attention head to obtain multiple candidate subsets. Then, a collaborative evaluation operation of inverse ranking fusion is performed on the candidate subsets to obtain the comprehensive retrieval score of each candidate subset.

[0058] For each candidate attention head subset, a document ranking list generated by each attention head is obtained, and the ranking is fused using a reciprocal ranking fusion method. After fusion, the retrieval score of the candidate subset on the calibration retrieval set is recalculated. Subsequently, a beam search method is used to progressively expand the candidate subset, retaining only the B (B is a positive integer) candidates with the highest comprehensive scores in each round, until the preset number of attention heads is reached, and finally, the binary attention mask of each layer is generated.

[0059] In operation S250, for each candidate subset, the new attention head addition operation, bundle search operation, and collaborative evaluation operation are iteratively performed until the preset conditions are met, the optimal subset is obtained, and the binary attention mask of each multi-head attention layer is generated using the optimal subset.

[0060] In operation S260, the semantic experts corresponding to each layer of the feedforward neural network are sparsified using binary neuron masks, and the multi-head attention layers are sparsified using binary attention masks, resulting in a sparsified dense retrieval model.

[0061] The binary neuron mask, binary attention mask, expert index table, and compression parameters generated at each layer are saved as a static configuration file. During the online inference phase, the relevant system loads this configuration file and skips the masked attention heads and feedforward network neurons during the Transformer forward propagation. For multi-head attention modules (i.e., multi-head attention layers), only the Q (query vector), K (key vector), and V (value vector) projections and attention outputs corresponding to the retained heads are calculated; for feedforward neural network layers, only the matrix slices corresponding to the retained neurons are calculated. The final dense retrieval model still outputs standard dense embedding vectors, which can be directly used in the relevant vector retrieval system.

[0062] The method provided in this application can achieve unstructured sparsity training for retrieval tasks without changing the weights of the original model, and reduce the inference computation overhead of dense embedding models.

[0063] The hybrid expert sparsity method for dense retrieval models provided in this application does not use internal statistics such as model weight amplitude and activation amplitude as selection criteria. Instead, it relies on the task activation frequency of semantic experts in the retrieval dataset to match the downstream retrieval needs of document recall. The entire process uses retrieval task metrics as the evaluation benchmark. First, a retrieval experience score based on recall metrics is obtained through single-head decoupling evaluation. Then, a reciprocal ranking fusion (RRF) method is used. RankFusion and the comprehensive retrieval score measure the effectiveness of the head combination. The selection criteria are fully aligned with the downstream retrieval targets of document recall, avoiding the accidental deletion of structural units that are crucial to retrieval ranking. This solves the problem of inconsistency between pruning criteria and downstream retrieval targets in existing technologies. At the same time, the hybrid expert sparsity method for dense retrieval models provided in this application does not require gradient backpropagation or model parameter update operations. The semantic experts of the feedforward neural network are obtained only by clustering the weight matrix of the original model. There is no need to fine-tune the router or retrain the expert allocation strategy. Activation frequency statistics, attention head evaluation, and bundle search optimization all only perform model forward propagation. Only a static binary mask is generated offline. The weights of the original dense retrieval model remain unchanged throughout the process. Structured sparsity is achieved by relying on the mask. Expert compression can be completed without additional training or fine-tuning. It can be directly adapted to the deployed encoding model, thus solving the problem of expert number compression relying on additional training in existing technologies. Furthermore, the hybrid expert sparsification method for dense retrieval models provided in this application abandons the approach of directly pruning based solely on single-head metrics. Instead, it first calculates the single-head retrieval score as an initial reference, without directly eliminating attention heads. It constructs a candidate subset of multiple attention heads, uses RRF to eliminate differences in scoring scales among different attention heads, quantifies the overall retrieval effect of the multi-head combination, it iteratively expands and evaluates the multi-head combination through bundle search, comprehensively considers the complementary and synergistic effects of multiple heads, and selects the optimal head combination based on the overall retrieval score of the subset. This approach takes into account the synergistic contributions among multiple attention heads, thereby solving the problem in existing technologies where attention head selection ignores the synergistic relationship among multiple heads.

[0064] Figure 3 An architecture diagram of a sparse hybrid expert compression scheme for dense retrieval according to an embodiment of this application is shown.

[0065] like Figure 3As shown, the technical solution provided in this application is a Retrieval-Oriented Sparse Mixture-of-Experts (RoS-MoE) scheme for dense retrieval. It aims to break the barriers of existing pruning methods where pruning criteria are misaligned with downstream retrieval targets and hybrid expertization schemes rely on large-scale retraining. It directly transforms "downstream retrieval results" into quantitative signals and implementation standards that can guide the sparsification of large-scale embedding models. This framework does not update any model parameters and mainly includes two stages: (1) FFN semantic expert partitioning and activation frequency masking, which performs task-related structured sparsity for the feedforward network layer; (2) retrieval result-driven attention head selection, which performs multi-head collaborative optimal subset search for the self-attention layer.

[0066] Among them, FFN expert sparse branches ( Figure 3 The green path shown: K-Means clustering is used to partition the feedforward network experts, and a mask is generated based on activation frequency to dynamically prune redundant semantic experts; among them, the attention head filtering branch ( Figure 3 (Blue path shown): Performs retrieval-driven heuristic dynamic clustering, conducts bundle search attention head selection using a fusion inverse ranking mechanism, and selects the optimal attention head combination. The two branches generate attention masks and neuron masks respectively, which work together to perform structured sparse processing on the original dense retrieval model, ultimately obtaining a sparse retrieval model.

[0067] According to an embodiment of this application, the above-described clustering process of each layer of the feedforward neural network of the dense retrieval model to obtain multiple semantic experts whose semantic boundaries do not overlap includes: setting an objective function for optimizing the structured semantic boundaries; and, based on the semantic activation mode, using the objective function to cluster the column vectors of the upprojection matrix in each layer of the feedforward neural network on the training dataset to obtain multiple semantic experts whose semantic boundaries do not overlap.

[0068] The semantic expert acquisition process provided in this application will be further described in detail below through specific implementation methods.

[0069] The basis of the technical solution in this application is that any standard Transformer embedding model can be structurally equivalently reorganized into a hybrid expert network by using the FFN layer in the manner of key-value caching (KV Cache), so that the weights do not need to be changed, and structured sparsity can be achieved simply by applying a binary mask to the network.

[0070] Let the first The upper projection matrix of the FFN layer is The lower projection matrix is The activation function is ,in, Represents a set of real numbers that have lines and Column matrix space, This represents the hidden layer dimension of the dense retrieval model. This represents the dimension of the intermediate layer of an FFN, i.e., the number of neurons contained in the intermediate layer of an FFN. The multi-head self-attention layer contains... Individual head An FFN neuron mask (i.e., a binary neuron mask) is introduced for each layer. Attention head mask (i.e., binary attention mask) The objective of this application is to achieve a preset retention rate. Under the constraints, find the factor that maximizes the downstream retrieval metrics (such as Recall@K, NDCG@K) for each layer. and .

[0071] The semantic expert segmentation and activation frequency masking process of FFN involved in this application first adopts a two-step strategy of "first clustering and then frequency filtering" to perform task-adaptive sparsification of the intermediate neurons of FFN.

[0072] Among them, FFN semantic expert partitioning: given the first Layer FFN , record its first Column vector is ,in, Indicates by Composed of real numbers A real vector space. Using K-Means clustering... The column vectors are clustered into Non-overlapping semantic experts (This specific implementation method takes) The optimization objective is shown in formula (1):

[0073] (1).

[0074] in For the first Cluster centers for experts. The above operations automatically divide the FFN interneurons into blocks according to semantic activation patterns, forming structured semantic boundaries.

[0075] According to an embodiment of this application, the above-mentioned method of calculating the average activation frequency of each semantic expert and comparing the average activation frequency with a preset retention rate to obtain active semantic experts and redundant semantic experts includes: traversing the model in a pure forward propagation manner, calculating the average activation frequency of each semantic expert on the task-related target training dataset, and obtaining the average activation frequency of each semantic expert; sorting each semantic expert in descending order according to the average activation frequency, and obtaining the descending order ranking result of the semantic experts; filtering the descending order ranking result of the semantic experts according to the preset retention rate to obtain active semantic experts, and defining other inactive semantic experts as redundant semantic experts.

[0076] According to an embodiment of this application, the above-mentioned statistical analysis of the average activation frequency of each semantic expert on the target training dataset related to the task includes: during the process of the dense retrieval model performing a gradient-free pure forward propagation operation on the current training sample in the target training dataset, obtaining the activation values ​​of all neurons in the current layer semantic expert after calculation by the activation function; using an indicator function to judge the activation values ​​to obtain the activation status of all neurons, and determining the activation proportion of the current layer semantic expert on the current training sample based on the activation status of all neurons; and processing the activation proportion of the current layer semantic expert on all training samples to obtain the average activation frequency of the current layer semantic expert on the target training dataset.

[0077] The process of task-related activation frequency statistics and mask generation involved in the above embodiments of this application will be further described in detail below through specific implementation methods.

[0078] In small-scale target datasets The model is traversed using pure forward propagation, for each expert... Average activation frequency across the entire dataset The statistics are as shown in formula (2):

[0079] (2).

[0080] in Input text length, This represents the activation function. For indicator functions, Representing text The Middle The word element in the first The corresponding input at layer FFN A 3D hidden state vector. Given a preset retention rate. ,according to Before sorting A subset of experts constitutes the activity. And based on this, generate an FFN binary mask. (Set the corresponding position of the activity expert to 1, and the rest to 0). During online reasoning, inject as in formula (3):

[0081] (3).

[0082] in, Indicates the first The input hidden state matrix of the FFN layer consists of the hidden state matrix corresponding to each word in the input text. Composed of 3D hidden state vectors, Indicates the first Layer application of FFN binary mask The resulting sparse feedforward neural network.

[0083] The shielded neurons can be indexed and sliced ​​out during matrix multiplication, thus truly reducing the amount of floating-point operations.

[0084] According to embodiments of this application, the above-described single-head decoupling evaluation operation performed on each attention head of each multi-head attention layer in the dense retrieval model to obtain the retrieval experience score of each attention head includes: isolating each attention head of each multi-head attention layer, retaining only one attention head as the data processing attention head during the forward propagation operation of the dense retrieval model; generating a query vector associated with the query text and a document vector associated with the document text using the data processing attention head, wherein the query text is obtained from the validation dataset and the document text is obtained from the document set; performing an inner product operation on the query vector and the document vector to obtain the semantic similarity between the query text and the document text; sorting the document set using the semantic similarity to obtain a document ranking list corresponding to each query text in the validation dataset; calculating the retrieval index of each query text using the document ranking list corresponding to each query text, and processing the retrieval indexes of all query texts to obtain the retrieval experience score of the data processing attention head.

[0085] The single-head decoupling evaluation operation involved in the above embodiments of this application will be further described in detail below through specific implementation methods.

[0086] To avoid the problems of misalignment of traditional attention head pruning metrics and neglect of multi-head collaboration, in the stage of attention head selection driven by retrieval results, downstream retrieval results are used as the sole evaluation signal to perform single-head evaluation, subset collaborative evaluation, and bundle search-style combination optimization on self-attention heads.

[0087] In the single-head decoupling evaluation process, the first... layer Each attention head is isolated individually, and only one head is retained for forward propagation to obtain the query text. With document text Origin Separately generated query vector representation Text vector representation And calculate single-head similarity using inner product. In a small validation set With document set The above calculates the retrieval experience score for each head. As shown in formula (4):

[0088] (4).

[0089] in, This indicates that only attention is retained. In the case of single-head similarity After sorting all candidate documents in document set C in descending order, the document retrieval sort list is obtained for the query text q.

[0090] According to an embodiment of this application, the above-mentioned beam search for each attention head based on retrieval experience score to obtain multiple candidate subsets includes: sorting the attention heads in descending order based on retrieval experience score to obtain a descending order sorting result of attention heads, and selecting multiple top-ranked attention heads from the descending order sorting result of attention heads according to a preset search beam width to obtain multiple candidate attention heads; and taking each candidate attention head as a candidate subset to obtain multiple candidate subsets.

[0091] According to embodiments of this application, the above-described collaborative evaluation operation of performing inverse ranking fusion on candidate subsets to obtain the comprehensive retrieval score for each candidate subset includes: generating a document ranking list corresponding to the query vector using each candidate attention head in the candidate subset, and obtaining the ranking value of each document text under each candidate attention head; calculating the contribution value of different candidate attention heads in the candidate subset to the ranking value for a single document text under the same query vector based on the set smoothing value; summing all contribution values ​​corresponding to a single document text to obtain the fusion score of a single document text; sorting all document texts in descending order based on the fusion score, and obtaining the comprehensive retrieval score of the candidate subset based on the result of the descending order sorting of document texts.

[0092] The following detailed description of the RRF-based subset collaboration evaluation process involved in the above embodiments of this application will be further explained through specific implementation methods.

[0093] Since the similarity scales of the outputs from different heads are different, Reciprocal Rank Fusion (RRF) is used to evaluate the candidate subsets. Merging multiple sorted lists yields the inverse ranking merged result. As shown in formula (5):

[0094] (5).

[0095] in, For attention head The given document text Integer ranking, This is the smoothing constant. (According to...) After reordering the documents, calculate the overall retrieval index corresponding to that subset. .

[0096] Meanwhile, in the process of selecting the optimal head subset through bundle search-based collaborative optimization, since the head combination space is... Since the magnitudes cannot be exhaustively enumerated, a beam search is used: Top The single head as the initial candidate subsets ( (for the bundle width); in each iteration, a new head is attempted to be added to each candidate subset, according to... Sort and keep only the first few One; when the size of the subset reaches Stop when the time is right, and output the optimal header subset. And generate an attention head mask based on this. .

[0097] According to embodiments of this application, the above-described hybrid expert sparsification method for dense retrieval models further includes: generating binary neuron masks and binary attention masks associated with the target task and target dataset offline without updating the original parameters of the dense retrieval model and without performing gradient backpropagation; receiving new query text online and generating effective output for sparse vector retrieval using the binary neuron masks and binary attention masks through online inference.

[0098] According to embodiments of this application, the above-mentioned online reception of new query text and generation of effective output for sparse vector retrieval using binary neuron masks and binary attention masks in an online reasoning manner includes: determining the active semantic expert of each feedforward neural network layer in the dense retrieval model based on the binary neuron mask; processing the new query text using the active semantic expert to obtain a standard dense embedding vector; determining the effective attention head in each multi-head attention layer in the dense retrieval model using the binary attention mask; projecting the query vector, value vector, and key vector generated by processing the standard dense embedding vector with the effective attention head to obtain the projection result of the effective attention head; and fusing the projection results of all effective attention heads to obtain the effective output of each multi-head attention layer.

[0099] The following detailed description of the "offline & online" two-stage data filtering process involved in this application will be further explained through specific implementation methods.

[0100] To integrate the two pathways of FFN semantic expert segmentation and activation frequency masking, as well as the attention head selection driven by retrieval results, into the inference process, this application designs a two-stage pipeline of offline and online.

[0101] (1) Offline stage: Perform K-Means clustering, activation frequency statistics and beam search on the target dataset to generate FFN masks for each layer. With attention head mask It is persisted as a lightweight static file and satisfies the zero-update parameter constraint. st Where st (such that) indicates that it is subject to, This represents the model parameters after offline processing. This represents the original model parameters before offline processing. This represents the learning rate or step size for parameter updates. The gradient operator represents the gradient with respect to the model parameter Θ. This represents the training loss function for the dense retrieval model, where all forward computations do not trigger backpropagation, and experiments are conducted in a non-training manner.

[0102] (2) Online phase: Receiving new query text The offline mask is then loaded into GPU memory, and the forward propagation of each Transformer layer is modified. Sparse multi-head attention is then applied. jump over The Q / K / V projection of the head is calculated as shown in formula (6):

[0103] (6).

[0104] in, Indicates the first The first in the layer The binary mask value corresponding to each attention head; when When = 1, the Q / K / V projection calculation of this attention head is retained and performed; when When =0, the attention head is blocked and its Q / K / V projection calculation is skipped; This represents the sparse multi-head self-attention operation after applying an attention head binary mask. This represents the scaled dot product attention operation, used to compute the output of a single attention head based on the query matrix, key matrix, and value matrix. Indicates the first The query projection matrix of each attention head. Indicates the first The key projection matrix of each attention head Indicates the first The output projection matrix of each attention head, Indicates the first The projection matrix of the values ​​of each attention head, Indicates the first The hidden state matrix output by the first Transformer layer, and also used as the... Input to the multi-head self-attention module.

[0105] Then, residuals and layer normalization are performed according to the standard Transformer structure, and then fed into the system. Controlled sparse FFN, iterating layer by layer until the th Layer. Final hidden state Average pooling along the sequence dimension yields dense embedding vectors. , Represents the query text The Middle The word element passes through the first The result after processing by the Transformer layer The hidden state vector is used; the output can then be used for standard vector retrieval. The method provided in this application directly uses the downstream retrieval results as a compression signal to drive sparsity, ultimately generating a sparse retrieval model that maintains retrieval quality at a sparsity rate of 50% and is fully compatible with existing RAG pipelines.

[0106] According to embodiments of this application, the dense retrieval models include dense retrieval models for biomedical retrieval, dense retrieval models for financial retrieval, dense retrieval models for fact-checking, dense retrieval models for entity retrieval, and dense retrieval models for long document retrieval.

[0107] To verify the effectiveness and robustness of the method provided in this application in large-scale dense retrieval model compression scenarios, experiments were conducted on 15 datasets of the Benchmark for Evaluation of Information Retrieval (BEIR), covering various downstream tasks such as fact checking, open-domain question answering, entity retrieval, biomedical retrieval, financial retrieval, and long document retrieval. Two representative large-scale embedding models were selected as the base models, and Recall@100 and NDCG@10 were used as the main quantitative evaluation metrics. To ensure sufficient comparison, the method provided in this application was compared with baseline methods such as unpruned dense models, Wanda pruning (a post-training pruning method for large language models), Magnitude-based Pruning (an unstructured pruning method sorted by absolute weight values), random pruning, and last-layer pruning. Through the above experimental setup, the technical effect of this application in dense retrieval model compression can be comprehensively evaluated from aspects such as recall, ranking quality, and performance preservation under different model architectures.

[0108] (1) The method provided in this application can effectively alleviate the problem of retrieval performance degradation that is prone to occur in traditional pruning methods.

[0109] Existing random pruning, weight-based pruning, and single-layer pruning methods typically compress parameters based on their statistical characteristics, and their pruning criteria are not entirely consistent with the semantic matching capabilities in downstream retrieval tasks. At high sparsity rates, these methods can easily mistakenly delete parameters that significantly contribute to the retrieval representation, leading to a significant decrease in retrieval performance. This application combines FFN expert activation frequency and retrieval result-driven attention head selection to make the pruning process more closely aligned with the needs of downstream retrieval tasks. Experimental results are shown in Tables 1 and 2. Under a 40% parameter pruning setting, the Recall@100 macro-average of the large-scale embedding model in this application reaches 0.7637, higher than the 0.5801 of the unpruned model, 0.5448 of Wanda pruning, 0.4449 of random pruning, and 0.5415 of the last-layer pruning. On the second type of large-scale embedding model, the average Recall@100 macro in this application reaches 0.6294, which is higher than 0.6051 for the unpruned model, 0.5805 for Wanda pruned models, 0.4365 for randomly pruned models, and 0.4864 for the last-layer pruned models. The above experimental results demonstrate that this application can maintain retrieval performance well while compressing model parameters, and achieves better results than the original dense model on some tasks. It should be noted that all datasets used in this application are open-source datasets; the use of these open-source datasets has been legally authorized, and the processing of these open-source datasets has been carried out in accordance with the requirements of relevant laws and regulations.

[0110] Among them, SciFact represents the scientific fact-checking dataset, which uses scientific assertions as queries and scientific paper abstracts as candidate texts to evaluate scientific evidence retrieval capabilities; NFCorpus represents NutritionFacts Corpus, a nutrition and medical information retrieval dataset, used to evaluate the model's ability to retrieve health questions and medical literature in natural language; ArguAna represents the argument and counter-argument retrieval dataset, used to retrieve the most relevant counter-arguments for a given argument; SCIDOCS represents the scientific document retrieval dataset, used to evaluate citation relationships between scientific papers and the ability to retrieve related literature; FiQA represents Financial Opinion Mining and Question Answering, used to evaluate text retrieval capabilities in the financial field; FEVER represents Fact Extraction and Verification, used to evaluate the model's ability to retrieve factual evidence from encyclopedic texts; MS MARCO represents Microsoft Machine Reading... Comprehension, the Microsoft Machine Reading Comprehension dataset, is used to evaluate the ability to match real user queries with candidate text paragraphs; HotpotQA represents a multi-hop question answering dataset, where multi-hop question answering refers to a question answering task that requires integrating information from multiple texts to obtain an answer; DBpedia-Entity represents an entity retrieval dataset built on the DBpedia knowledge base, used to retrieve relevant entities based on natural language queries; Climate-FEVER represents a climate fact extraction and verification dataset, used to evaluate the ability to retrieve evidence and verify facts related to climate change; NQ represents Natural Questions, a dataset consisting of real user query questions and their corresponding encyclopedia texts, used to evaluate open-domain question answering and text retrieval capabilities; Quora represents a question pair dataset based on the Quora question answering community, used to evaluate the ability to identify duplicate questions and retrieve semantically similar questions; Multi-News represents a multi-document news dataset, consisting of multiple related reports and their summaries, used to evaluate the semantic representation and retrieval capabilities of news texts; Touché-2020 represents Touché The 2020 Argument Retrieval dataset is used to retrieve relevant arguments based on controversial issues; MacroAvg. represents the macro average, which is the unweighted arithmetic average of the results across all datasets after calculating Recall@100 for each dataset separately.

[0111] Table 1: Recall@100 for large-scale embedding models of the first class on various datasets

[0112]

[0113] Table 2: Recall@100 for large-scale embedding models of type II on various datasets

[0114]

[0115] (2) This application can achieve model compression without training, thereby reducing deployment costs.

[0116] Existing methods for transforming dense models into sparse or hybrid expert models typically require additional fine-tuning, further training, or sparse upgrade processes, resulting in high training costs and hindering direct deployment in existing retrieval systems. In contrast, this application only requires forward propagation on the target dataset in the offline phase, calculating the activation frequencies of FFN experts, and selecting a superior attention head combination based on the retrieval results, without needing to update the gradients of the original model parameters. This application can generate lightweight mask files while maintaining the original model weights, achieving training-free model compression. This approach reduces additional training overhead and facilitates direct application to already deployed large-scale dense embedding models.

[0117] (3) This application is compatible with existing dense search and RAG pipelines.

[0118] This application applies masking control only to a portion of the FFN experts and attention heads during the Transformer's forward propagation phase in the online inference stage, and the final output remains a standard dense vector. Therefore, this application does not require changes to the overall structure of existing vector retrieval systems and can continue to interface with FAISS (an open-source, industrial-grade vector retrieval library), Approximate Nearest Neighbor (ANN) indexes, vector databases, and upper-layer RAG application processes. It not only reduces redundant computation during model inference but also maintains good engineering compatibility with existing dense retrieval systems, making it suitable for model compression and deployment optimization in existing retrieval enhancement generation systems, enterprise knowledge base retrieval systems, and multi-domain semantic search systems.

[0119] This application primarily addresses the technical problems existing in large-scale dense embedding models when performing sparse compression for retrieval tasks. These problems include inconsistencies between pruning criteria and downstream retrieval targets, reliance on additional training for expert-level compression, and neglect of multi-head collaboration in attention head selection. These issues lead to decreased retrieval performance at higher compression rates, make it difficult to apply the compression process to deployed models at low cost, and result in insufficient task adaptability of the compressed model.

[0120] First, this paper addresses the technical problem of inconsistent pruning criteria with downstream metrics in dense retrieval, leading to decreased retrieval performance at high compression rates. Existing pruning methods typically determine parameter importance based on metrics such as weight magnitude, activation statistics, gradient sensitivity, or second-order approximation information. However, these metrics primarily reflect internal model parameters or activation states and do not directly correspond to retrieval ranking metrics such as Recall@K, NDCG@K, and Mean Reciprocal Rank (MRR). Therefore, pruning based on these metrics may inadvertently delete parameters that play a crucial role in query-document semantic discrimination or retain redundant parameters that contribute little to the retrieval results, thereby weakening the discriminative ability of the vector space.

[0121] Secondly, it addresses the technical problem that existing hybrid expert partitioning methods rely on additional training, making them difficult to apply cost-effectively to already deployed dense embedding models. Existing MoE partitioning methods typically require continued training, router fine-tuning, or relearning expert allocation strategies, resulting in high engineering and computational costs, making them difficult to directly apply to already trained and deployed models. While some training-free expert partitioning methods reduce training costs, their partitioning is often based on parameter clustering, activation patterns, or token-level statistical features, without directly utilizing downstream retrieval results for feedback. This can easily disrupt the relative distance relationship between query vectors and document vectors, affecting the differentiation of similar documents.

[0122] Furthermore, this paper addresses the technical issue of existing attention head pruning methods neglecting multi-head collaboration relationships, leading to a mismatch between attention layer compression results and downstream retrieval performance. Existing attention head pruning methods typically rank and prune individual attention heads based on attention entropy, L1 norm, gradient sensitivity, or single-head ablation results. However, different attention heads may have complementary and collaborative relationships, and the importance of a single attention head does not fully represent its contribution to the overall head combination. Therefore, pruning based solely on single-head statistics easily overlooks the impact of multi-head combinations on the final retrieval ranking results.

[0123] Based on the above-described hybrid expert sparsification method for dense retrieval models, embodiments of this application also provide a hybrid expert sparsification apparatus for dense retrieval models. The following will be combined with... Figure 4 The device is described in detail.

[0124] Figure 4 A structural block diagram of a hybrid expert sparsification apparatus for a dense retrieval model according to an embodiment of this application is shown.

[0125] like Figure 4As shown, the hybrid expert sparsification device 400 for dense retrieval models in this embodiment includes a semantic clustering module 410, a neuron mask generation module 420, a single-head decoupling evaluation module 430, a reciprocal ranking fusion module 440, an attention mask generation module 450, and a model sparsification module 460.

[0126] The semantic clustering module 410 is used to perform clustering processing on each layer of the feedforward neural network of the dense retrieval model to obtain multiple semantic experts whose semantic boundaries do not overlap. The dense retrieval model is used to encode the query text and document text input by the user into dense semantic vectors and use the dense semantic vectors stored in the memory to complete the task of recalling the user-specified document. In one embodiment, the semantic clustering module 410 can be used to execute the step S210 described above, which will not be repeated here.

[0127] The neuron mask generation module 420 is used to calculate the average activation frequency of each semantic expert, compare the average activation frequency with a preset retention rate to obtain active semantic experts and redundant semantic experts, and use the active semantic experts and redundant semantic experts to generate a binary neuron mask for each layer of the feedforward neural network. In one embodiment, the neuron mask generation module 420 can be used to perform the step S220 described above, which will not be repeated here.

[0128] The single-head decoupling evaluation module 430 is used to perform a single-head decoupling evaluation operation on each attention head of each multi-head attention layer in the dense retrieval model to obtain the retrieval experience score of each attention head. In one embodiment, the single-head decoupling evaluation module 430 can be used to execute the step S230 described above, which will not be repeated here.

[0129] The inverse ranking fusion module 440 is used to perform a beam search on each attention head based on the retrieval experience score to obtain multiple candidate subsets, and to perform a collaborative evaluation operation of inverse ranking fusion on the candidate subsets to obtain a comprehensive retrieval score for each candidate subset. In one embodiment, the inverse ranking fusion module 440 can be used to execute the step S240 described above, which will not be repeated here.

[0130] The attention mask generation module 450 is used to iteratively perform new attention head addition operation, bundle search operation and collaborative evaluation operation on each candidate subset until the preset conditions are met to obtain the optimal subset, and use the optimal subset to generate a binary attention mask for each multi-head attention layer; in one embodiment, the attention mask generation module 450 can be used to perform the step S250 described above, which will not be repeated here.

[0131] The model sparsification module 460 is used to sparsify the semantic experts corresponding to each layer of the feedforward neural network using binary neuron masks and to sparsify each multi-head attention layer using binary attention masks, thereby obtaining a sparsified dense retrieval model. In one embodiment, the model sparsification module 460 can be used to execute step S260 described above, which will not be repeated here.

[0132] According to embodiments of this application, any multiple modules among the semantic clustering module 410, neuron mask generation module 420, single-head decoupling evaluation module 430, inverse ranking fusion module 440, attention mask generation module 450, and model sparsity module 460 can be merged into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of this application, at least one of the semantic clustering module 410, neuron mask generation module 420, single-head decoupling evaluation module 430, inverse ranking fusion module 440, attention mask generation module 450, and model sparsity module 460 can be at least partially implemented as hardware circuits, such as field-programmable gate arrays, programmable logic arrays, systems-on-a-chip, systems-on-a-substrate, systems-on-package, application-specific integrated circuits, or any other reasonable means of integrating or packaging circuits, or implemented in software, hardware, or firmware, or in any appropriate combination of any of these three implementation methods. Alternatively, at least one of the semantic clustering module 410, neuron mask generation module 420, single-head decoupling evaluation module 430, inverse ranking fusion module 440, attention mask generation module 450, and model sparsification module 460 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0133] Figure 5 A block diagram of an electronic device suitable for implementing a hybrid expert sparsification method for dense retrieval models, according to an embodiment of this application, is shown.

[0134] like Figure 5As shown, an electronic device 500 according to an embodiment of this application includes a processor 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory 502 or a program loaded from a storage portion 508 into a random access memory 503. The processor 501 may include, for example, a general-purpose microprocessor, an instruction set processor and / or an associated chipset and / or a dedicated microprocessor. The processor 501 may also include onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for executing different steps of the method flow according to an embodiment of this application.

[0135] Random access memory 503 stores various programs and data required for the operation of electronic device 500. Processor 501, read-only memory 502, and random access memory 503 are interconnected via bus 504. Processor 501 executes various steps of the method flow according to embodiments of this application by executing programs in read-only memory 502 and / or random access memory 503. It should be noted that the programs may also be stored in one or more memories other than read-only memory 502 and random access memory 503. Processor 501 may also execute various steps of the method flow according to embodiments of this application by executing programs stored in said one or more memories.

[0136] According to embodiments of this application, the electronic device 500 may further include an input / output interface 505, which is also connected to a bus 504. The electronic device 500 may also include one or more of the following components connected to the input / output interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a cathode ray tube, liquid crystal display, etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card, such as a local area network card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the input / output interface 505 as needed. A removable medium 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 510 as needed so that computer programs read from it can be installed into the storage section 508 as needed.

[0137] Embodiments of this application also provide a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0138] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, portable compact disk read-only memory, optical storage devices, magnetic storage devices, or any suitable combination thereof. In embodiments of this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this application, the computer-readable storage medium may include the read-only memory 502, and / or random access memory 503, and / or one or more memories other than read-only memory 502 and random access memory 503 described above.

[0139] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the methods provided in the embodiments of this application.

[0140] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 509, and / or installed from a removable medium 511. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0141] In embodiments of this application, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by processor 501, it performs the functions defined in the system of this application embodiment. According to embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0142] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0143] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0144] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

Claims

1. A hybrid expert sparsification method for dense retrieval models, characterized in that, The method includes: Each layer of the feedforward neural network in the dense retrieval model is clustered to obtain multiple semantic experts whose semantic boundaries do not overlap. The dense retrieval model is used to encode the query text and document text input by the user into dense semantic vectors and use the dense semantic vectors stored in memory to complete the task of recalling the user-specified document. The average activation frequency of each semantic expert is calculated, and the average activation frequency is compared with a preset retention rate to obtain active semantic experts and redundant semantic experts. The active semantic experts and the redundant semantic experts are then used to generate a binary neuron mask for each layer of the feedforward neural network. A single-head decoupling evaluation operation is performed on each attention head of each multi-head attention layer in the dense retrieval model to obtain the retrieval experience score of each attention head; Based on the retrieval experience score, a beam search is performed on each attention head to obtain multiple candidate subsets, and a reciprocal ranking fusion collaborative evaluation operation is performed on the candidate subsets to obtain the comprehensive retrieval score of each candidate subset. For each candidate subset, the new attention head addition operation, bundle search operation, and collaborative evaluation operation are performed iteratively until a preset condition is met to obtain the optimal subset, and the optimal subset is used to generate the binary attention mask for each multi-head attention layer. The binary neuron mask is used to sparsify the semantic expert corresponding to each layer of the feedforward neural network, and the binary attention mask is used to sparsify each layer of the multi-head attention layer, to obtain a sparsified dense retrieval model.

2. The method according to claim 1, characterized in that, Clustering is performed on each layer of the feedforward neural network in the dense retrieval model to obtain multiple semantic experts whose semantic boundaries do not overlap, including: Set the objective function for optimizing the structured semantic boundaries; Based on the semantic activation pattern, on the training dataset, the column vectors of the upprojection matrix in each layer of the feedforward neural network are clustered using the objective function to obtain multiple semantic experts whose semantic boundaries do not overlap.

3. The method according to claim 1, characterized in that, The average activation frequency of each semantic expert is calculated and compared with a preset retention rate to obtain active semantic experts and redundant semantic experts, including: The model is traversed using pure forward propagation, and the average activation frequency of each semantic expert on the task-related target training dataset is statistically analyzed to obtain the average activation frequency of each semantic expert. The semantic experts are sorted in descending order based on the average activation frequency to obtain the descending order ranking result of the semantic experts; Based on the preset retention rate, the descending sorting results of the semantic experts are filtered to obtain the active semantic experts, and other inactive semantic experts are defined as the redundant semantic experts.

4. The method according to claim 3, characterized in that, The statistical analysis of the average activation frequency of each semantic expert on the task-related target training dataset includes: During the process of the dense retrieval model performing a gradient-free pure forward propagation operation on the current training samples in the target training dataset, the activation values ​​of all neurons in the current layer semantic expert are obtained after calculation by the activation function. The activation value is judged using an indicator function to obtain the activation status of all neurons, and the activation ratio of the current layer semantic expert on the current training sample is determined based on the activation status of all neurons. The activation percentage of the current layer semantic expert on all training samples is processed to obtain the average activation frequency of the current layer semantic expert on the target training dataset.

5. The method according to claim 1, characterized in that, For each attention head in each multi-head attention layer of the dense retrieval model, a single-head decoupling evaluation operation is performed to obtain the retrieval experience score for each attention head, including: Each attention head of the multi-head attention layer is isolated, and only one attention head is retained as the data processing attention head during the forward propagation operation of the dense retrieval model. The data processing attention head is used to generate a query vector associated with the query text and a document vector associated with the document text, wherein the query text is obtained from the validation dataset and the document text is obtained from the document set; The semantic similarity between the query text and the document text is obtained by performing an inner product operation on the query vector and the document vector. The document set is sorted using the semantic similarity to obtain a sorted list of documents corresponding to each query text in the verification dataset; The retrieval index of each query text is calculated using the document sorting list corresponding to each query text, and the retrieval indexes of all query texts are processed to obtain the retrieval experience score of the data processing attention.

6. The method according to claim 1, characterized in that, Based on the retrieval experience score, a beam search is performed on each attention head to obtain multiple candidate subsets, including: The attention heads are sorted in descending order based on the retrieval experience score to obtain the descending order ranking result of the attention heads. Based on the preset search beam width, multiple top-ranked attention heads are selected from the descending order ranking result of the attention heads to obtain multiple candidate attention heads. Each candidate attention head is used as a candidate subset to obtain multiple candidate subsets.

7. The method according to claim 1, characterized in that, A collaborative evaluation operation of reciprocal ranking fusion is performed on the candidate subsets to obtain a comprehensive retrieval score for each candidate subset, including: Using each candidate attention head in the candidate subset, a document ranking list corresponding to the query vector is generated, and the ranking value of each document text under each candidate attention head is obtained; Based on the set smoothing value, for a single document text under the same query vector, calculate the contribution value of different candidate attention heads in the candidate subset to the ranking value; The fusion score of a single document text is obtained by summing up all the contribution values ​​corresponding to the single document text. Based on the fusion score, all document texts are sorted in descending order, and the comprehensive retrieval score of the candidate subset is obtained based on the result of the descending sort of the document texts.

8. The method according to any one of claims 1 to 7, characterized in that, Also includes: Without updating the original parameters of the dense retrieval model and without performing gradient backpropagation, generate binary neuron masks and binary attention masks associated with the target task and target dataset offline. New query text is received online, and the binary neuron mask and the binary attention mask are used to generate an effective output for sparse vector retrieval in an online reasoning manner.

9. The method according to claim 8, characterized in that, Receiving new query text online, and generating effective output for sparse vector retrieval using the binary neuron mask and the binary attention mask in an online reasoning manner, including: The active semantic expert of each layer of the feedforward neural network in the dense retrieval model is determined based on the binary neuron mask, and the new query text is processed using the active semantic expert to obtain a standard dense embedding vector; The effective attention heads in each multi-head attention layer of the dense retrieval model are determined using the binary attention mask. The query vector, value vector, and key vector generated by processing the standard dense embedding vector with the effective attention heads are then projected to obtain the projection result of the effective attention heads. The projection results of all the effective attention heads are fused to obtain the effective output of each multi-head attention layer.

10. The method according to any one of claims 1 to 7, characterized in that, The dense retrieval models include dense retrieval models for biomedical retrieval, dense retrieval models for financial retrieval, dense retrieval models for fact-checking, dense retrieval models for entity retrieval, and dense retrieval models for long document retrieval.

Images