Geometric perception pruning and sparsification method for enterprise knowledge document-oriented large language model

Abstract

This invention discloses a geometric-aware pruning and sparsity method and apparatus for large language models of enterprise knowledge documents, relating to the field of large language model compression and lightweighting technology. The method includes: acquiring data from enterprise private knowledge documents and database records; constructing a real calibration sample dataset to perform forward inference on an uncompressed large language model, generating two sets of independently perturbed hidden states; generating corresponding intermediate layer outputs based on the two sets of independently perturbed hidden states; quantifying the difference between the two outputs using matrix difference entropy to obtain the sensitivity of each layer's perturbation; performing adaptive hierarchical sparsity allocation based on the sensitivity of each layer's perturbation, converting the sensitivity of each layer's perturbation into the degree of contribution of that layer to the semantic preservation of the enterprise knowledge document; and converting the contribution of each layer into a pruning rate to complete the pruning. This invention can solve the problem of knowledge omissions when large language models process long enterprise documents.

Description

Technical Field

[0001] This invention relates to the field of large language model compression and lightweighting technology, and in particular to a geometric-aware pruning and sparsity method and apparatus for large language models of enterprise knowledge documents. Background Technology

[0002] With the rapid development of artificial intelligence technology, large language models (LLMs) have become a core technology in the field of natural language processing. These models typically employ the Transformer architecture and, through large-scale pre-training and parameter expansion, demonstrate powerful performance in text generation, complex reasoning, and semantic understanding. For example, models such as OpenAI's GPT series and Meta's LLaMA series have hundreds of billions or even trillions of parameters, resulting in enormous computational costs: the ultra-large scale of parameters leads to extremely high floating-point operations during inference, heavily relying on high-performance GPU clusters, and the model weights and activation values ​​during inference consume massive amounts of GPU memory, limiting their deployment on mobile or edge computing devices. All of these factors lead to a sharp increase in computational costs and memory consumption, limiting their deployment on resource-constrained devices (such as mobile devices or edge computing). To address this issue, model compression techniques have emerged, mainly including three typical schemes: quantization, knowledge distillation, and weight pruning.

[0003] Quantization reduces memory footprint by lowering the precision of weights and activations. A representative method is GPTQ, which uses second-order Hessian information to achieve efficient quantization, but may introduce quantization errors. Knowledge distillation, through a teacher-student framework, uses soft labels output by a large model to guide the training of a lightweight smaller model, transferring knowledge from the large model to the smaller one, as exemplified by TinyBERT. While this method performs well, it requires redesigning and retraining the model architecture for different scenarios, resulting in high development costs. Weight pruning is the most direct compression method, inducing sparsity by removing redundant parameters, thereby reducing model size and accelerating inference. Pruning can be divided into unstructured pruning and structured pruning. Early pruning methods often used uniform sparse distribution, meaning all Transformer layers applied the same sparsity ratio. However, due to inter-layer heterogeneity—shallow networks primarily capture local syntactic features and require retaining more neurons, while mid-to-deep networks handle complex semantic logic and long-range dependencies—neurons in deeper networks begin to exhibit task-related structures, and uniform distribution significantly degrades performance under high sparsity. Therefore, non-uniform pruning has become a mainstream trend, using hierarchical adaptive allocation of sparsity ratios to prioritize the protection of critical layers and achieve performance preservation at higher compression ratios. Overall, development in this field has shifted from simple parameter removal to data-driven and structure-aware compression, aiming to balance model efficiency and accuracy. However, existing solutions often rely on indirect heuristics, ignoring the intrinsic geometric structure of hidden representations, leading to poor semantic stability under extreme sparsity, necessitating more accurate robustness evaluation methods.

[0004] Existing LLM pruning techniques are mainly divided into two categories: uniform sparse allocation and non-uniform sparse allocation. These techniques aim to compress model size and accelerate inference by removing redundant parameters, but most rely on indirect indicators such as weight magnitude, activation of outliers, or search algorithms, failing to directly quantify the geometric robustness of layers to semantic perturbations, leading to significant performance degradation under high sparsity conditions. Uniform sparse pruning methods apply the same sparsity ratio across all layers, with the core assumption that the sensitivity to parameter removal is consistent across layers. The typical SparseGPT method works by optimizing weights block by block using a small calibration dataset, reconstructing the pruned weights using approximate second-order information to ensure the output is close to the original model. The drawback is that the uniform allocation strategy ignores the heterogeneity between LLM layers. Different layers contribute significantly to model performance and their robustness to parameter changes; forcing a uniform sparsity ratio can lead to a drastic performance drop under extremely high sparsity conditions.

[0005] Existing non-uniform pruning methods primarily rely on heuristic statistical features of weights or activations to assess the importance of network layers. They fail to characterize the pruning sensitivity of network layers from the perspective of the geometric structure and semantic stability of hidden representations within the distribution of real-world enterprise private data (internal documents, knowledge bases, structured databases, historical records, compliance clauses, and domain-specific terminology). Furthermore, existing techniques generally ignore the characteristic that hidden representations in knowledge-intensive enterprise scenarios are often highly concentrated in low-dimensional manifolds, making it difficult to distinguish the different impacts of semantic preservation perturbations and destructive perturbations on model behavior. This can easily introduce systematic biases into hierarchical pruning decisions, leading to significant knowledge omissions, semantic drift, or illusion risks when the model handles long enterprise documents, cross-document association reasoning, and multi-turn knowledge question answering or compliance retrieval tasks. In addition, existing techniques often use simple distance metrics or loss changes as evaluation indicators for hierarchical sensitivity measurement, lacking a universally applicable metric tool with information theory significance that can stably quantify changes in hidden representation structure. This makes it difficult to accurately reflect the differences in the amplification effect of perturbations among different network layers, thus limiting the applicability and robustness of non-uniform pruning strategies under extremely high sparsity conditions. Summary of the Invention

[0006] To address the shortcomings of existing technologies, such as neglecting the characteristic that hidden representations in knowledge-intensive enterprise scenarios are often highly concentrated in low-dimensional manifolds, making it difficult to distinguish the different impacts of semantic preservation perturbations and destructive perturbations on model behavior, and thus easily introducing systematic biases in hierarchical pruning decisions, leading to significant knowledge omissions, semantic drift, or illusion risks when the model handles long enterprise documents, cross-document association reasoning, multi-turn knowledge question answering, or compliance retrieval tasks, this invention provides a geometric-aware pruning and sparsity method and apparatus for large language models of enterprise knowledge documents. The technical solution is as follows: On the one hand, a geometry-aware pruning and sparsity method for large language models of enterprise knowledge documents is provided. This method is implemented by a geometry-aware pruning and sparsity device for large language models of enterprise knowledge documents, and includes: S1. Obtain data from enterprise private knowledge documents and database records, and construct a real calibration sample dataset; S2. Use real calibration sample datasets to perform forward inference on the uncompressed large language model and extract the intermediate hidden states of the large language model under the real data distribution of the enterprise. S3. The hidden state of the intermediate layer is mapped to a low-dimensional tangent space using principal component analysis to obtain a low-dimensional representation. A controlled perturbation is added to the low-dimensional representation to construct a perturbed representation. The perturbed representation is then mapped back to the original feature space through back projection to generate two sets of independent perturbed hidden states. S4. Process the hidden states after the first set of independent perturbations through the weight matrix of the corresponding intermediate layer to obtain the first output; process the hidden states after the second set of independent perturbations through the weight matrix of the corresponding intermediate layer to obtain the second output; based on the first output and the second output, use matrix difference entropy to quantify the difference between the two outputs to obtain the sensitivity of each Transformer layer perturbation in the large language model. S5. Based on the sensitivity of the perturbation of each Transformer layer, perform adaptive hierarchical sparse allocation, convert the sensitivity of the perturbation of each Transformer layer into the degree of contribution of each Transformer layer to the semantic preservation of enterprise knowledge documents; convert the degree of contribution of each layer into the pruning rate, complete the pruning and output the pruned large language model; deploy the pruned large language model on the enterprise's internal computing devices to provide intelligent services for the enterprise's private knowledge documents and database records.

[0007] On the other hand, a geometry-aware pruning and sparsity management device for large language models of enterprise knowledge documents is provided. This device is applied to the geometry-aware pruning and sparsity management method for large language models of enterprise knowledge documents. The device includes: The data acquisition unit is used to acquire data from enterprise proprietary knowledge documents and database records, and to construct a realistic calibration dataset; The extraction unit is used to perform forward inference on an uncompressed large language model using a real calibration dataset to extract the intermediate hidden states of the large language model under the real data distribution of the enterprise. The generation unit is used to map the hidden state of the intermediate layer to a low-dimensional tangent space using principal component analysis to obtain a low-dimensional representation; to add a controlled perturbation to the low-dimensional representation to construct a perturbed representation; and to map the perturbed representation back to the original feature space through back projection to generate two sets of independent perturbed hidden states. The sensitivity calculation unit is used to process the hidden states after the first set of independent perturbations through the weight matrix of the corresponding intermediate layer to obtain the first output; to process the hidden states after the second set of independent perturbations through the weight matrix of the corresponding intermediate layer to obtain the second output; based on the first output and the second output, the difference between the two outputs is quantified by matrix difference entropy to obtain the sensitivity of the perturbation of each Transformer layer in the large language model. The sparse allocation and pruning execution unit is used to perform adaptive hierarchical sparse allocation based on the sensitivity of each Transformer layer to perturbations. It converts the sensitivity of each Transformer layer to perturbations into the degree of contribution of each Transformer layer to the semantic preservation of enterprise knowledge documents. It converts the contribution of each layer into a pruning rate, completes pruning, and outputs the pruned large language model. The pruned large language model is deployed on enterprise internal computing devices to provide intelligent services for enterprise private knowledge documents and database records.

[0008] On the other hand, a large language model geometry-aware pruning and sparsity management device for enterprise knowledge documents is provided. The device includes: a processor; a memory storing computer-readable instructions, which, when executed by the processor, implement any of the methods described above for large language model geometry-aware pruning and sparsity management for enterprise knowledge documents.

[0009] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the at least one instruction being loaded and executed by a processor to implement any of the above-described methods of geometrically aware pruning and sparsity methods for large language models of enterprise knowledge documents.

[0010] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: This invention models the local manifold structure of hidden representations under the real data distribution of enterprise proprietary knowledge documents and databases. Under strict semantic preservation constraints, it adds geometrically perturbations to accurately assess the importance of each network layer from the perspectives of geometric stability and semantic consistency of enterprise proprietary knowledge, avoiding the manifold deviation from semantic drift caused by traditional isotropic perturbations. By adding a matrix difference entropy metric with clear information theory significance, the differences in hidden representations before and after perturbation are quantified. This invention can stably and objectively characterize the sensitivity differences of different network layers to semantic preservation perturbations of enterprise proprietary knowledge documents, achieving interpretable assessment of layer robustness. Furthermore, it adaptively generates differentiated non-uniform sparse distribution ratios to ensure that even under high sparsity conditions, the model can still effectively retain its ability to understand and generate core enterprise business knowledge (such as internal reports, compliance documents, domain-specific terminology, or historical records). This significantly reduces problems such as increased illusions, inaccurate answers, knowledge omissions, or compliance risks caused by excessive compression in private deployment environments (local servers, private clouds, and edge database appliances). Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart of a geometric-aware pruning and sparsity method for large language models of enterprise knowledge documents provided in an embodiment of the present invention; Figure 2 This is a flowchart of a neural network pruning and sparsity allocation based on hidden manifold geometric stability provided by an embodiment of the present invention; Figure 3 This is a block diagram of a geometric perception pruning and sparsity device for a large language model for enterprise knowledge documents provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a geometric perception pruning and sparse device for a large language model for enterprise knowledge documents provided in an embodiment of the present invention. Detailed Implementation

[0013] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0014] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0015] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0016] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0017] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0018] This invention provides a geometry-aware pruning and sparsity method for large language models of enterprise knowledge documents. This method can be implemented using a geometry-aware pruning and sparsity device for large language models of enterprise knowledge documents, which can be a terminal or a server. Figure 1 The flowchart shown is for a geometry-aware pruning and sparsity method for large language models of enterprise knowledge documents. The processing flow of this method may include the following steps: S1. Obtain data from the enterprise's private knowledge documents and database records, and construct a real calibration sample dataset.

[0019] The data recorded in the enterprise's private knowledge documents and databases includes: internal reports, knowledge base entries, and structured data records.

[0020] in, Figure 2 This is a flowchart of a neural network pruning and sparsity allocation based on hidden manifold geometric stability provided by an embodiment of the present invention.

[0021] We use real calibration sample datasets to perform forward inference on an uncompressed large language model and extract the intermediate hidden states of the large language model under the real data distribution of the enterprise.

[0022] S2. Using real calibration sample datasets, perform forward inference on the uncompressed large language model to extract the intermediate hidden states of the large language model under the real data distribution of the enterprise.

[0023] S3. Principal component analysis is used to map the hidden states of the intermediate layer to a low-dimensional tangent space to obtain a low-dimensional representation. A controlled perturbation is added to the low-dimensional representation to construct a perturbed representation. The perturbed representation is then mapped back to the original feature space through back projection to generate two sets of independently perturbed hidden states.

[0024] Optionally, S3 uses principal component analysis to map the hidden states of the intermediate layers to a low-dimensional tangent space to obtain a low-dimensional representation, including: S31. Transfer each input calibration sample dataset to the large language model. The hidden activation state vector of the corresponding layer is denoted as The hidden activation state vector is then centered to obtain the corresponding mean vector. In this embodiment of the invention, the hidden activation state vector refers to the intermediate representation corresponding to each knowledge document sample after the input sequence has been processed by each layer of the large language model. These vectors are usually obtained from the outputs.hidden_states list of the model. They carry the model's encoding of semantic information at different depths and are observed to be located on a low-dimensional manifold, which is the basis for geometric perturbation and layer sensitivity estimation in this embodiment of the invention.

[0025] S32. Based on the corresponding mean vector, principal component analysis is used to retain K principal components to construct an orthogonal basis, and the hidden state representation is mapped to a low-dimensional tangent space to obtain a low-dimensional representation. The process of obtaining the low-dimensional representation is represented by the following formula (1): (1) in, Indicates the first The hidden state representation corresponding to the layer; Represents the mean vector; Indicates an orthogonal basis; This represents a low-dimensional representation.

[0026] Optionally, the process of adding a controlled perturbation to the low-dimensional representation to construct the perturbated representation, and then mapping the perturbated representation back to the original feature space through back projection, is represented by the following formulas (2)-(3): (2) (3) in, Indicates the representation after perturbation; This represents a low-dimensional representation; Indicates a controlled disturbance; Indicates an orthogonal basis; Mean vector; This represents the hidden state vector that has been put back into the high-dimensional space after geometric perturbation and is available for processing by the next layer; This represents the i-th knowledge document sample.

[0027] The perturbation is confined to the local tangent space to ensure that it unfolds along the dominant change direction of the hidden representation, thereby maintaining the semantic continuity of the enterprise knowledge document and avoiding invalid changes that deviate from the manifold due to isotropic noise.

[0028] Depending on the enterprise's deployment needs, such as processing sensitive knowledge documents on private database servers or edge devices, the forms of disturbance can include various types.

[0029] Optionally, the disturbance may take the form of tangential noise, Brownian noise, or geodetic noise. Among them, tangential noise is used to characterize local linear robustness by injecting Gaussian noise into the tangential space, and is used to evaluate the stability of the model to local semantic variations of enterprise documents. In one feasible implementation, the tangential noise is represented by the following formula (4): (4) in, Indicates tangential noise; This represents a low-dimensional representation; It is a small, isotropic Gaussian noise vector applied in a low-dimensional tangent space, which serves to simulate the natural drift of activation on the semantic manifold, thereby probing the layer’s sensitivity to semantic preservation perturbations. It represents the variance of the perturbation applied to the low-dimensional tangent space, which controls the energy intensity of the noise and determines the strength of the sensitivity of the "detector" layer; It is a k×k identity matrix and is the covariance matrix of a Gaussian distribution.

[0030] Brown noise, in particular, enhances the ability to assess the stability of long-sequence knowledge documents by accumulating random perturbations along the sequence dimension to simulate time-series correlation drift. In one feasible implementation, Brownian noise is represented by the following formula (5): (5) in, This represents the new coordinates of the t-th token in the tangent space after Brownian motion perturbation; This represents the coordinates of the t-th token in the original sequence in the low-dimensional tangent space; This indicates accumulation along the sequence.

[0031] Among them, geodesic noise adaptively adjusts the perturbation amplitude by combining the scale information of each principal component direction, and is used to generate semantically preserving perturbations that fit the manifold curvature characteristics of enterprise knowledge documents, so as to ensure the geometric consistency of private data in the database deployment model.

[0032] In one feasible implementation, geodetic noise is represented by the following formula (6): (6) in, Indicates tangential noise; This represents a low-dimensional representation; It is a small, isotropic Gaussian noise vector applied in a low-dimensional tangent space, which serves to simulate the natural drift of activation on the semantic manifold, thereby probing the layer’s sensitivity to semantic preservation perturbations. This represents the empirical standard deviation along each principal component direction.

[0033] Among them, by using local tangent space approximation and geometric perception perturbation generation process, multiple sets of mutated activations can be obtained without destroying the semantic structure of enterprise knowledge documents. This provides a basis for subsequent accurate evaluation of the network layer's response characteristics to perturbations within the manifold, thereby supporting the efficient pruning and sparse deployment of large language models in enterprise private environments.

[0034] S4. Process the hidden states after the first set of independent perturbations through the weight matrix of the corresponding intermediate layer to obtain the first output; process the hidden states after the second set of independent perturbations through the weight matrix of the corresponding intermediate layer to obtain the second output; based on the first output and the second output, use matrix difference entropy to quantify the difference between the two outputs to obtain the sensitivity of each layer of perturbation.

[0035] Optionally, the process of processing the hidden states after the first set of independent perturbations through the weight matrix of the corresponding intermediate layer to obtain the first output is represented by the following formula (7): (7) in, This represents the output activation generated by the Lth layer after forward propagation of the first perturbation version of the i-th calibration sample; This represents the module matrix of the l-th layer of the Transformer; This represents the hidden state after the first set of perturbations; The process of processing the hidden states after the second set of independent perturbations through the weight matrix of the corresponding intermediate layer to obtain the second output is represented by the following formula (8): (8) in, This represents the output activation generated by forward propagation of the second perturbation version of the i-th calibration sample in the L-th layer; This represents the module matrix of the l-th layer of the Transformer; This represents the hidden state after the second set of perturbations.

[0036] Optionally, based on the first and second outputs, the process of quantifying the difference between the two outputs using matrix difference entropy to obtain the sensitivity of each layer to perturbations is expressed by the following formula (9): (9) in, Represent two high-dimensional activation sets and The degree of difference in geometric structure between them is used to quantify the sensitivity of a certain layer of a large language model to small perturbations; This indicates activation by the output after the first set of perturbations. The constructed kernel matrix is ​​used for measurement. The similarity between samples in the layer is used to capture the geometric characteristics of the output distribution of that layer. Renyi-α entropy, which represents matrix-based entropy, measures the amount of information or complexity contained in a kernel matrix K. This is represented as the average of the joint geometric entropy after randomly shuffling the sample order, which serves as a baseline for comparing the geometric structural similarity between two output sets, thereby extracting the structural information brought about by the true correspondence. This represents the kernel matrix constructed from the output activations after the second set of perturbations; it is used to measure... The similarity between samples in the layer is used to capture the geometric characteristics of the output distribution of that layer.

[0037] Among them, the sensitivity index of disturbance directly measures the first After receiving semantic preservation perturbations along the tangent space of the enterprise knowledge document manifold, the layer measures the degree of damage to the output distribution structure. Low entropy indicates robustness, i.e., low sparsity in the distribution; high entropy indicates fragility, i.e., high sparsity in the distribution. This provides an interpretable geometric sensitivity index and avoids the limitations of heuristics.

[0038] S5. Based on the sensitivity of each layer's perturbation, perform adaptive hierarchical sparse allocation, converting the sensitivity of each layer's perturbation into the degree of contribution of that layer to the semantic preservation of enterprise knowledge documents; convert the degree of contribution of each layer into the pruning rate, and complete the pruning.

[0039] Each Transformer layer in the large language model contains an attention sublayer and a feedforward network sublayer, with residual connections and layer normalization before and after each sublayer.

[0040] Optionally, S5's adaptive hierarchical sparsity allocation based on the sensitivity of each Transformer layer perturbation, transforming the sensitivity of each Transformer layer perturbation into the degree of contribution of each Transformer layer to the semantic preservation of enterprise knowledge documents, is represented by the following formula (10): (10) in, Indicates the first The degree of contribution of the geometric robustness of the layer; This represents the maximum value of the perturbation sensitivity of each layer; This indicates the sensitivity of each layer to disturbances; Among them, the lower the contribution level, that is, the lower the sensitivity of the layer to perturbations on the semantic manifold, the higher the contribution of the layer to semantic preservation.

[0041] The process of converting the contribution level of each layer into the pruning rate is represented by the following formulas (11)-(13): (11) (12) (13) in, Presentation layer Normalized robustness score; Parameters representing control deviation; Indicates the first The degree of contribution of the geometric robustness of the layer; Indicates the degree of geometric robustness contribution in the Transformer layer. The minimum value; Indicates the degree of geometric robustness contribution in the Transformer layer. The maximum value; express The arithmetic mean, which serves as a global baseline offset, is used to ensure that the final inter-layer sparsity adjustment does not change the globally set average sparsity target during non-uniform sparsity distribution. This represents the total number of Transformer layers in a large language model LLM; Indicates the global target sparsity specified by the user; Indicates assignment to the first The actual pruning rate of each Transformer layer.

[0042] The user-specified global target sparsity is typically given by enterprises based on the memory / computing resource limitations of the deployed hardware, such as 20%, 30%, or 40%. These parameters control the deviation. Make the pruning rate fluctuate at Between these values, the sparsity rate of a single layer is limited to limit extreme deviations, preventing catastrophic performance degradation in tasks such as enterprise real-world question answering, knowledge retrieval, or document generation due to excessive pruning of key semantic layers.

[0043] In one feasible implementation, the pruned large language model can be deployed on enterprise-internal computing devices to provide intelligent services for enterprise-owned knowledge documents and database records. Specific applications include: Enterprise internal knowledge base Q&A: Employees ask questions using natural language, and the model accurately retrieves answers from internal reports, technical documents, and operation manuals. After pruning, the inference speed is improved and the memory usage is reduced.

[0044] Compliance review and risk control: Automatically analyze contracts, compliance clauses, and audit reports to identify potential risk points. Even after pruning, the model remains sensitive to technical terms and logical relationships, reducing the risk of compliance omissions.

[0045] Intelligent summarization and information extraction for long documents: For long documents such as project reports and research literature, generate concise summaries or extract key information to help employees quickly grasp the core content.

[0046] Cross-document relational reasoning: It integrates knowledge from multiple documents to answer questions that require cross-document relational reasoning (such as "the conflict points between a certain project and a certain specification"). Pruning preserves the semantic manifold structure and ensures the consistency of reasoning.

[0047] Private deployment ensures data security: the model runs entirely within the enterprise, and sensitive data does not need to be uploaded to the cloud, meeting data privacy and compliance requirements while reducing reliance on external computing power.

[0048] The pruning method proposed in this invention compresses the model while preserving the semantic integrity of key enterprise knowledge, enabling large language models to efficiently and accurately support enterprise knowledge management and intelligent interaction on resource-constrained hardware.

[0049] In one feasible implementation, the adaptive hierarchical sparse allocation method adopted in this invention can prioritize the protection of robust layers and pruning-vulnerable layers with semantic perturbations, thereby achieving performance improvement under high pruning rates. This significantly enhances the consistency, accuracy, and interpretability of large language models for sensitive knowledge content under high sparsity in enterprise private deployment environments (such as local servers, private clouds, and edge database appliances), providing a more reliable lightweight large model solution for core business scenarios such as document intelligent question answering, compliance retrieval, and internal knowledge base enhancement for knowledge-intensive enterprises.

[0050] This invention models the local manifold structure of hidden representations under the real data distribution of enterprise proprietary knowledge documents and databases. Under strict semantic preservation constraints, it adds geometrically perturbations to accurately assess the importance of each network layer from the perspectives of geometric stability and semantic consistency of enterprise proprietary knowledge, avoiding the manifold deviation from semantic drift caused by traditional isotropic perturbations. By adding a matrix difference entropy metric with clear information theory significance, the differences in hidden representations before and after perturbation are quantified. This invention can stably and objectively characterize the sensitivity differences of different network layers to semantic preservation perturbations of enterprise proprietary knowledge documents, achieving interpretable assessment of layer robustness. Furthermore, it adaptively generates differentiated non-uniform sparse distribution ratios to ensure that even under high sparsity conditions, the model can still effectively retain its ability to understand and generate core enterprise business knowledge (such as internal reports, compliance documents, domain-specific terminology, or historical records). This significantly reduces problems such as increased illusions, inaccurate answers, knowledge omissions, or compliance risks caused by excessive compression in private deployment environments (local servers, private clouds, and edge database appliances).

[0051] Figure 3This is a block diagram of a geometry-aware pruning and sparsity management device for large language models of enterprise knowledge documents, provided in an embodiment of the present invention. This device is used for a geometry-aware pruning and sparsity management method for large language models of enterprise knowledge documents. (Refer to...) Figure 3 The device includes a data acquisition unit 310, an extraction unit 320, a generation unit 330, a sensitivity calculation unit 340, and a sparse allocation and pruning execution unit 350. Wherein: The data acquisition unit 310 is used to acquire data from enterprise private knowledge documents and database records, and to construct a real calibration dataset; Extraction unit 320 is used to perform forward inference on an uncompressed large language model using a real calibration dataset to extract the intermediate hidden states of the large language model under the real data distribution of the enterprise. The generation unit 330 is used to map the hidden state of the intermediate layer to a low-dimensional tangent space using principal component analysis to obtain a low-dimensional representation; to add a controlled perturbation to the low-dimensional representation to construct a perturbed representation; and to map the perturbed representation back to the original feature space through back projection to generate two sets of independent perturbed hidden states. Sensitivity calculation unit 340 is used to process the hidden states after the first set of independent perturbations through the weight matrix of the corresponding intermediate layer to obtain the first output; process the hidden states after the second set of independent perturbations through the weight matrix of the corresponding intermediate layer to obtain the second output; based on the first output and the second output, the difference between the two outputs is quantified by matrix difference entropy to obtain the sensitivity of the perturbation of each Transformer layer in the large language model. The sparse allocation and pruning execution unit 350 is used to perform adaptive hierarchical sparse allocation based on the sensitivity of the perturbation of each Transformer layer, converting the sensitivity of the perturbation of each Transformer layer into the degree of contribution of each Transformer layer to the semantic preservation of enterprise knowledge documents; converting the contribution of each layer into the pruning rate, completing the pruning and outputting the pruned large language model; and deploying the pruned large language model on the enterprise's internal computing devices to provide intelligent services for the enterprise's private knowledge documents and database records.

[0052] Optionally, the generation unit 330 includes: Each input calibration sample dataset is used in the large language model. The hidden activation state vector of the corresponding layer is denoted as The hidden activation state vector is then centered to obtain the corresponding mean vector. Based on the corresponding mean vector, principal component analysis is used to retain K principal components to construct an orthogonal basis, and the hidden state representation is mapped to a low-dimensional tangent space to obtain a low-dimensional representation. The process of obtaining the low-dimensional representation is represented by the following formula (1): (1) in, Indicates the first The hidden state representation corresponding to the layer; Represents the mean vector; Indicates an orthogonal basis; This represents a low-dimensional representation.

[0053] Optionally, the process of adding a controlled perturbation to the low-dimensional representation, constructing the perturbated representation, and mapping the perturbated representation back to the original feature space through back projection is represented by the following formulas (2)-(3): (2) (3) in, Indicates the representation after perturbation; This represents a low-dimensional representation; Indicates a controlled disturbance; Indicates an orthogonal basis; Mean vector; This represents the hidden state vector that has been returned to the high-dimensional space after geometric perturbation and is available for processing by the next layer; This represents the i-th knowledge document sample.

[0054] Optionally, the disturbance may take the form of tangential noise, Brownian noise, or geodesic noise. Among them, tangential noise is used to characterize local linear robustness by injecting Gaussian noise into the tangential space, and is used to evaluate the stability of the model to local semantic variations of enterprise documents. Brown noise, in particular, enhances the ability to assess the stability of long-sequence knowledge documents by accumulating random perturbations along the sequence dimension to simulate time-series correlation drift. Among them, geodesic noise adaptively adjusts the perturbation amplitude by combining the scale information of each principal component direction, and is used to generate semantically preserving perturbations that fit the manifold curvature characteristics of enterprise knowledge documents, so as to ensure the geometric consistency of private data in the database deployment model.

[0055] Optionally, the process of processing the first set of independently perturbed hidden states through the weight matrix of the corresponding intermediate layer to obtain the first output is represented by the following formula (4): (4) in, This represents the output activation generated by the Lth layer after forward propagation of the first perturbation version of the i-th calibration sample; This represents the module matrix of the l-th layer of the Transformer; This represents the hidden state after the first set of perturbations; The process of processing the hidden states after the second set of independent perturbations through the weight matrix of the corresponding intermediate layer to obtain the second output is represented by the following formula (5): (5) in, This represents the output activation generated by forward propagation of the second perturbation version of the i-th calibration sample in the L-th layer; This represents the module matrix of the l-th layer of the Transformer; This represents the hidden state after the second set of perturbations.

[0056] Optionally, the process of obtaining the sensitivity of each layer's perturbation by quantifying the difference between the first and second outputs using matrix difference entropy is represented by the following formula (6): (6) in, Represent two high-dimensional activation sets and The degree of difference in geometric structure between them is used to quantify the sensitivity of a certain layer of a large language model to small perturbations; This indicates activation by the output after the first set of perturbations. The constructed kernel matrix is ​​used for measurement. The similarity between samples in the layer is used to capture the geometric characteristics of the output distribution of that layer. Renyi-α entropy, which represents matrix-based entropy, measures the amount of information or complexity contained in a kernel matrix K. This is represented as the average of the joint geometric entropy after randomly shuffling the sample order, which serves as a baseline for comparing the geometric structural similarity between two output sets, thereby extracting the structural information brought about by the true correspondence. This represents the kernel matrix constructed from the output activations after the second set of perturbations; it is used to measure... The similarity between samples in the layer is used to capture the geometric characteristics of the output distribution of that layer.

[0057] Optionally, the process of adaptively allocating hierarchical sparsity based on the sensitivity of perturbation of each Transformer layer, and converting the sensitivity of perturbation of each Transformer layer into the degree of contribution of each Transformer layer to the semantic preservation of enterprise knowledge documents, is represented by the following formula (7): (7) in, Indicates the first The degree of contribution of the geometric robustness of the layer; This represents the maximum value of the perturbation sensitivity of each layer; This indicates the sensitivity of each layer to disturbances; Represents the first... layer; The process of converting the contribution level of each layer into the pruning rate is represented by the following formulas (8)-(10): (8) (9) (10) in, Presentation layer Normalized robustness score; Parameters representing control deviation; Indicates the first The degree of contribution of the geometric robustness of the layer; Indicates the degree of geometric robustness contribution in the Transformer layer. The minimum value; Indicates the degree of geometric robustness contribution in the Transformer layer. The maximum value; express The arithmetic mean, which serves as a global baseline offset, is used to ensure that the final inter-layer sparsity adjustment does not change the globally set average sparsity target during non-uniform sparsity distribution. This represents the total number of Transformer layers in a large language model LLM; Indicates the global target sparsity specified by the user; Indicates assignment to the first The actual pruning rate of each Transformer layer.

[0058] This invention models the local manifold structure of hidden representations under the real data distribution of enterprise proprietary knowledge documents and databases. Under strict semantic preservation constraints, it adds geometrically perturbations to accurately assess the importance of each network layer from the perspectives of geometric stability and semantic consistency of enterprise proprietary knowledge, avoiding the manifold deviation from semantic drift caused by traditional isotropic perturbations. By adding a matrix difference entropy metric with clear information theory significance, the differences in hidden representations before and after perturbation are quantified. This invention can stably and objectively characterize the sensitivity differences of different network layers to semantic preservation perturbations of enterprise proprietary knowledge documents, achieving interpretable assessment of layer robustness. Furthermore, it adaptively generates differentiated non-uniform sparse distribution ratios to ensure that even under high sparsity conditions, the model can still effectively retain its ability to understand and generate core enterprise business knowledge (such as internal reports, compliance documents, domain-specific terminology, or historical records). This significantly reduces problems such as increased illusions, inaccurate answers, knowledge omissions, or compliance risks caused by excessive compression in private deployment environments (local servers, private clouds, and edge database appliances).

[0059] Figure 4 This is a schematic diagram of the structure of a geometry-aware pruning and sparsity device for large language models of enterprise knowledge documents provided in an embodiment of the present invention, as shown below. Figure 4 As shown, the geometry-aware pruning and sparsity device for large language models of enterprise knowledge documents can include the above-mentioned... Figure 3 The illustrated large language model geometry-aware pruning and sparsity device for enterprise knowledge documents. Optionally, the large language model geometry-aware pruning and sparsity device 410 for enterprise knowledge documents may include a first processor 2001.

[0060] Optionally, the large language model geometry-aware pruning and sparse device 410 for enterprise knowledge documents may also include a memory 2002 and a transceiver 2003.

[0061] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0062] The following is combined Figure 4 A detailed introduction is given to the various components of the large language model geometry-aware pruning and sparsity device 410 for enterprise knowledge documents: The first processor 2001 is the control center of the large language model geometry-aware pruning and sparsity device 410 for enterprise knowledge documents. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0063] Optionally, the first processor 2001 can execute various functions of the large language model geometry-aware pruning and sparse device 410 for enterprise knowledge documents by running or executing software programs stored in memory 2002 and calling data stored in memory 2002.

[0064] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 4 CPU0 and CPU1 are shown in the diagram.

[0065] In a specific implementation, as one example, the SSS device 410 may also include multiple processors, for example... Figure 4 The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor or a multi-core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0066] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0067] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently, and may be connected via the interface circuit of the large language model geometry-aware pruning and sparse device 410 for enterprise knowledge documents. Figure 4 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0068] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0069] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 4 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0070] Optionally, the transceiver 2003 can be integrated with the first processor 2001 or exist independently, and can be connected to the interface circuit of the large language model geometry-aware pruning and sparse device 410 for enterprise knowledge documents. Figure 4 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0071] It should be noted that, Figure 4 The structure of the large language model geometry-aware pruning and sparse device 410 for enterprise knowledge documents shown in the figure does not constitute a limitation on the router. The actual large language model geometry-aware pruning and sparse device for enterprise knowledge documents may include more or fewer components than shown, or combine some components, or have different component arrangements.

[0072] Furthermore, the technical effects of the large language model geometry-aware pruning and sparsity device 410 for enterprise knowledge documents can be referred to the technical effects of the large language model geometry-aware pruning and sparsity method for enterprise knowledge documents described in the above method embodiments, and will not be repeated here.

[0073] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or it may be any conventional processor, etc.

[0074] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0075] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0076] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0077] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0078] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0079] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0080] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0081] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0082] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0083] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0084] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0085] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An enterprise knowledge document-oriented large language model geometric perception pruning and sparsification method, characterized in that, The method includes: S1. Obtain data from enterprise private knowledge documents and database records, and construct a real calibration sample dataset; S2. Use real calibration sample datasets to perform forward inference on the uncompressed large language model and extract the intermediate hidden states of the large language model under the real data distribution of the enterprise. S3. The hidden state of the intermediate layer is mapped to a low-dimensional tangent space using principal component analysis to obtain a low-dimensional representation. A controlled perturbation is added to the low-dimensional representation to construct a perturbed representation. The perturbed representation is then mapped back to the original feature space through back projection to generate two sets of independent perturbed hidden states. S4. Process the hidden states after the first set of independent perturbations through the weight matrix of the corresponding intermediate layer to obtain the first output; process the hidden states after the second set of independent perturbations through the weight matrix of the corresponding intermediate layer to obtain the second output; based on the first output and the second output, use matrix difference entropy to quantify the difference between the two outputs to obtain the sensitivity of each Transformer layer perturbation in the large language model. S5. Based on the sensitivity of the perturbation of each Transformer layer, perform adaptive hierarchical sparse allocation, convert the sensitivity of the perturbation of each Transformer layer into the degree of contribution of each Transformer layer to the semantic preservation of enterprise knowledge documents; convert the degree of contribution of each layer into the pruning rate, complete the pruning and output the pruned large language model; deploy the pruned large language model on the enterprise's internal computing devices to provide intelligent services for the enterprise's private knowledge documents and database records.

2. The enterprise knowledge document-oriented large language model geometric perception pruning and sparsification method according to claim 1, characterized in that, The S3 method uses principal component analysis to map the hidden states of the intermediate layer to a low-dimensional tangent space to obtain a low-dimensional representation, including: S31. Transfer each input calibration sample dataset to the large language model. The hidden activation state vector of the corresponding layer is denoted as The hidden activation state vector is then centered to obtain the corresponding mean vector. S32. Based on the corresponding mean vector, principal component analysis is used to retain K principal components to construct an orthogonal basis, and the hidden state representation is mapped to a low-dimensional tangent space to obtain a low-dimensional representation. The process of obtaining the low-dimensional representation is represented by the following formula (1): （1） in, Indicates the first The hidden state representation corresponding to the layer; Represents the mean vector; Indicates an orthogonal basis; This represents a low-dimensional representation.

3. The geometry-aware pruning and sparsity method for large language models of enterprise knowledge documents according to claim 1, characterized in that, The process of adding a controlled perturbation to the low-dimensional representation, constructing the perturbated representation, and mapping the perturbated representation back to the original feature space through back projection is represented by the following formulas (2)-(3): （2） （3） in, Indicates the representation after perturbation; This represents a low-dimensional representation; Indicates a controlled disturbance; Indicates an orthogonal basis; Mean vector; This represents the hidden state vector that has been returned to the high-dimensional space after geometric perturbation and is available for processing by the next layer; This represents the i-th knowledge document sample.

4. The geometry-aware pruning and sparsity method for large language models of enterprise knowledge documents according to claim 1, characterized in that, The forms of the disturbance include: tangential noise, Brownian noise, and geodesic noise; Among them, tangential noise is used to characterize local linear robustness by injecting Gaussian noise into the tangential space, and is used to evaluate the stability of the model to local semantic variations of enterprise documents. Brown noise, in particular, enhances the ability to assess the stability of long-sequence knowledge documents by accumulating random perturbations along the sequence dimension to simulate time-series correlation drift. Among them, geodesic noise adaptively adjusts the perturbation amplitude by combining the scale information of each principal component direction, and is used to generate semantically preserving perturbations that fit the manifold curvature characteristics of enterprise knowledge documents, so as to ensure the geometric consistency of private data in the database deployment model.

5. The geometry-aware pruning and sparsity method for large language models of enterprise knowledge documents according to claim 1, characterized in that, The process of processing the hidden states after the first set of independent perturbations through the weight matrix of the corresponding intermediate layer to obtain the first output is represented by the following formula (4): （4） in, This represents the output activation generated by the Lth layer after forward propagation of the first perturbation version of the i-th calibration sample; This represents the module matrix of the l-th layer of the Transformer; This represents the hidden state after the first group of perturbations; The process of processing the hidden states after the second set of independent perturbations through the weight matrix of the corresponding intermediate layer to obtain the second output is represented by the following formula (5): （5） in, This represents the output activation generated by forward propagation of the second perturbation version of the i-th calibration sample in the L-th layer; This represents the module matrix of the l-th layer of the Transformer; This indicates the hidden state after the second group of perturbations.

6. The geometry-aware pruning and sparsity method for large language models of enterprise knowledge documents according to claim 1, characterized in that, The process of obtaining the sensitivity of each layer to perturbation by quantifying the difference between the first and second outputs using matrix difference entropy is expressed by the following formula (6): （6） in, Represent two high-dimensional activation sets and The degree of difference in geometric structure between them is used to quantify the sensitivity of a certain layer of a large language model to small perturbations; This indicates activation by the output after the first set of perturbations. The constructed kernel matrix is ​​used for measurement. The similarity between samples in the layer is used to capture the geometric characteristics of the output distribution of that layer. Renyi-α entropy, which represents matrix-based entropy, measures the amount of information or complexity contained in a kernel matrix K. This is represented as the average of the joint geometric entropy after randomly shuffling the sample order, which serves as a baseline for comparing the geometric structural similarity between two output sets, thereby extracting the structural information brought about by the true correspondence. This represents the kernel matrix constructed from the output activations after the second set of perturbations; it is used to measure... The similarity between samples in the layer is used to capture the geometric characteristics of the output distribution of that layer.

7. The geometry-aware pruning and sparsity method for large language models of enterprise knowledge documents according to claim 1, characterized in that, The adaptive hierarchical sparse allocation based on the sensitivity of each Transformer layer perturbation in S5, which converts the sensitivity of each Transformer layer perturbation into the degree of contribution of each Transformer layer to the semantic preservation of enterprise knowledge documents, is represented by the following formula (7): （7） in, Indicates the first The degree of contribution of the geometric robustness of the layer; This represents the maximum value of the perturbation sensitivity of each layer; This indicates the sensitivity of each layer to disturbances; The process of converting the contribution level of each layer into the pruning rate is represented by the following formulas (8)-(10): （8） （9） （10） in, Presentation layer Normalized robustness score; Parameters representing control deviation; Indicates the first The degree of contribution of the geometric robustness of the layer; Indicates the degree of geometric robustness contribution in the Transformer layer. The minimum value; Indicates the degree of geometric robustness contribution in the Transformer layer. The maximum value; express The arithmetic mean, which serves as a global baseline offset, is used to ensure that the final inter-layer sparsity adjustment does not change the globally set average sparsity target during non-uniform sparsity distribution. This represents the total number of Transformer layers in a large language model LLM; Indicates the global target sparsity specified by the user; Indicates assignment to the first The actual pruning rate of each Transformer layer.

8. A geometry-aware pruning and sparsity management device for large language models of enterprise knowledge documents, wherein the device is used to implement the geometry-aware pruning and sparsity management method for large language models of enterprise knowledge documents as described in any one of claims 1-7, characterized in that... The device includes: The data acquisition unit is used to acquire data from enterprise proprietary knowledge documents and database records, and to construct a realistic calibration dataset; The extraction unit is used to perform forward inference on an uncompressed large language model using a real calibration dataset to extract the intermediate hidden states of the large language model under the real data distribution of the enterprise. The generation unit is used to map the hidden state of the intermediate layer to a low-dimensional tangent space using principal component analysis to obtain a low-dimensional representation; to add a controlled perturbation to the low-dimensional representation to construct a perturbed representation; and to map the perturbed representation back to the original feature space through back projection to generate two sets of independent perturbed hidden states. The sensitivity calculation unit is used to process the hidden states after the first set of independent perturbations through the weight matrix of the corresponding intermediate layer to obtain the first output; to process the hidden states after the second set of independent perturbations through the weight matrix of the corresponding intermediate layer to obtain the second output; based on the first output and the second output, the difference between the two outputs is quantified by matrix difference entropy to obtain the sensitivity of the perturbation of each Transformer layer in the large language model. The sparse allocation and pruning execution unit is used to perform adaptive hierarchical sparse allocation based on the sensitivity of each Transformer layer to perturbations. It converts the sensitivity of each Transformer layer to perturbations into the degree of contribution of each Transformer layer to the semantic preservation of enterprise knowledge documents; it converts the contribution of each layer into a pruning rate, completes pruning, and outputs the pruned large language model; it deploys the pruned large language model on enterprise internal computing devices to provide intelligent services for enterprise private knowledge documents and database records.

9. A geometry-aware pruning and sparsity control device for large language models of enterprise knowledge documents, characterized in that, The geometry-aware pruning and sparsity control device for large language models of enterprise knowledge documents includes: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 7.

Images