A model layering distillation and enhancement method and system for audit verticals

Abstract

The application discloses a model layering distillation and enhancement method and system for audit vertical fields, relates to the technical field of audit field model distillation, and first acquires a law and tax audit field pre-training large language model as a teacher model, selects a lightweight small language model with parameter quantity meeting the end side computing power requirement as a to-be-trained student model, constructs a law and tax audit training sample set labeled with five reasoning level results, calls the teacher model to generate soft labels of each level, obtains constraint weights of each level through a pre-training expert score calculator, trains the student model by adopting a hybrid constraint strategy covering intermediate reasoning representation, evidence reference and final output, and outputs a usable enhanced model after end side adaptation verification. The application improves the reasoning accuracy and full-link compliance of the lightweight law and tax audit model, reduces the deployment computing power threshold, adapts to various law and tax audit business scenarios, and improves the efficiency of first-line audit work.

Description

Technical Field

[0001] This invention relates to the field of model distillation technology in the auditing field, and in particular to a method and system for hierarchical model distillation and enhancement oriented towards the vertical field of auditing. Background Technology

[0002] Tax and legal auditing services cover multiple scenarios, including tax risk verification, legal compliance review, and financial audit assessment. The process requires multi-layered reasoning and judgment on massive amounts of business documents, electronic documents, and financial vouchers, demanding high accuracy, compliance, and interpretability of the reasoning results. While current large-scale language models have achieved reasoning accuracy approaching that of professionals in tax and legal auditing, their high parameter count and computational resource requirements far exceed the capacity of edge devices such as tax field terminals and enterprise financial audit terminals, making direct deployment in frontline business scenarios impossible. Lightweight small-scale language models, although with lower parameter counts and adaptable to edge devices, suffer from poor reasoning accuracy when directly trained, and the compliance of the reasoning chain fails to meet the professional requirements of tax and legal auditing, making direct application in actual business difficult.

[0003] Current mainstream solutions for improving the performance of small models fall into two main categories. The first is the conventional knowledge distillation approach. This approach uses the soft labels output by a teacher model with a large number of parameters as supervisory signals to train a student model with a smaller number of parameters. This improves the inference accuracy of the student model without increasing its parameter count. This approach has been widely used in general natural language processing scenarios and can effectively reduce the dependence of small model training on labeled data. However, this approach only constrains the final output of the student model and does not cover the intermediate steps of model inference. In scenarios such as legal and tax auditing, which require compliance through end-to-end reasoning, it is prone to issues where the student model's final output is correct, but the intermediate reasoning logic is flawed, or the evidence cited does not meet professional requirements, thus failing to meet business compliance requirements. The second category is vertical domain small model fine-tuning solutions. This approach directly uses labeled data from the vertical domain to perform supervised fine-tuning of a lightweight model. The trained small model can adapt to the basic inference needs of the vertical domain. However, this approach requires a large amount of professional domain labeled data with extremely high annotation costs, and the small model struggles to learn complex multi-layered inference logic, resulting in a significant gap in the professionalism of the inference results compared to larger models.

[0004] Existing knowledge distillation solutions, when adapted to professional vertical fields such as law, taxation, and auditing, do not consider the differences in importance of different reasoning stages and cannot assign corresponding constraint weights to the professional requirements of each reasoning stage. The small models trained are prone to problems such as insufficient accuracy in core reasoning stages and redundant constraints in non-core stages. At the same time, existing solutions do not impose separate constraints on the evidence citation stage of the student model, resulting in poor interpretability of the student model's output results. This fails to meet the requirements of law, taxation, and auditing businesses for traceable and verifiable reasoning chains, thus limiting the application of lightweight models in front-line business scenarios of law, taxation, and auditing. Summary of the Invention

[0005] This invention proposes a hierarchical distillation and enhancement method and system for models in the vertical field of auditing, to solve the problems mentioned in the prior art. The technical solution is as follows: In a first aspect, embodiments of this application provide a model hierarchical distillation and enhancement method for the vertical field of auditing, including: We acquire a pre-trained large language model in the fields of law, taxation, and auditing as the teacher model, and a lightweight small language model with fewer than a preset threshold of parameters as the student model to be trained. Construct a hierarchical labeled training sample set for legal and tax auditing. Each sample in the sample set corresponds to a labeling result at five levels: fact identification, rule matching, element judgment, risk attribution, and disposal suggestion. The teacher model is invoked to perform full-link inference on all samples in the training sample set, and outputs the soft label results corresponding to the five levels of each sample. At the same time, the pre-trained expert scorer is invoked to score the quality of the output results of the teacher model at each level and the hard label results corresponding to the stratification of the samples, so as to obtain the weight coefficients corresponding to the five levels. The inference chain of the student model is divided into five levels of inference branches. Each inference branch receives the sample input features of the corresponding level and outputs the inference result of the corresponding level. A hybrid constraint strategy is used to iteratively train the student model. The hybrid constraint simultaneously covers the deviation constraints between the three types of objects in the student model—intermediate inference representations at each level of the student model, evidence citation results, and final output results—and the corresponding output results of the teacher model, as well as the hard label results of sample stratification. The trained student model is validated for edge-side adaptability using a pre-labeled legal and tax audit validation set, and an enhanced student model that meets edge-side deployment requirements is output.

[0006] Furthermore, the total distillation loss during training is calculated using a stratified distillation loss function weighted by stratified expert scores, the function expression of which is: ;in This represents the total distillation loss during student model training. This represents the weighting coefficient obtained from the expert score corresponding to the i-th reasoning level. This represents the preset weighting coefficients for the intermediate inference loss. Let represent the mean squared error loss of the intermediate inference representation of the student model at the i-th inference level compared to the corresponding output of the teacher model. The pre-defined weighting coefficients representing the loss of evidence citation. Let represent the cross-entropy loss between the evidence citation entries output by the student model at the i-th reasoning level and the corresponding outputs of the teacher model, as well as the hard-labeled citation entries for sample stratification. This represents the preset weighting coefficients for the final output loss. This represents the KL divergence loss between the output of the student model at the i-th inference level and the hard labels of the sample hierarchy and the soft labels of each level of the teacher model.

[0007] Furthermore, the construction process of the pre-trained expert scorer is as follows: a hierarchical reasoning quality scoring dataset labeled by senior experts in the fields of law, taxation, and auditing is pre-collected. The dataset covers three core scenarios: tax risk verification, legal compliance review, and financial audit judgment. It includes three types of scoring results: accuracy, compliance, and relevance, corresponding to output samples of different reasoning levels. A lightweight text evaluation model is used to conduct supervised training on the scoring dataset. After training, the expert scorer automatically outputs a hierarchical weight coefficient in the range of 0 to 1 for any input hierarchical reasoning output result. The value of the hierarchical weight coefficient is positively correlated with the quality of the corresponding hierarchical reasoning output result.

[0008] Furthermore, the process of constructing a hierarchically labeled training sample set for legal and tax audits is as follows: original business documents, electronic documents, financial vouchers, and field business verification records from three scenarios—tax risk verification, legal compliance review, and financial audit judgment—are collected as the original sample inputs for hierarchical labeling. For each sample input, the following are labeled sequentially: entity and event extraction results at the fact identification level; matching legal provisions and policy entries at the rule matching level; compliance judgment results at the requirement judgment level; risk point location and cause analysis results at the risk attribution level; and rectification and response plan results at the disposal suggestion level.

[0009] Furthermore, the five-level inference branches of the student model are built using an architecture that shares a bottom-level feature extraction layer and consists of independent upper-level task heads. The bottom-level feature extraction layer is completed using a lightweight Transformer encoder. The number of attention heads and the dimension of the hidden layers in the encoder are preset according to the computing power threshold of the target device. Each layer of independent task heads sets an appropriate output layer structure for the task type of the corresponding level. The fact recognition level task head is set to a sequence labeling output structure, the rule matching level task head is set to a multi-label classification output structure, the element judgment level task head is set to a binary classification output structure, the risk attribution level task head is set to a controllable text generation output structure, and the disposal suggestion level task head is set to a controllable text generation output structure.

[0010] Furthermore, during the execution of the hybrid constraint strategy, the constraints on the intermediate inference representation specifically involve feature alignment between the output of the last hidden layer of each hierarchical inference branch of the student model and the hidden layer output of the corresponding hierarchical inference link of the teacher model. The constraints on the evidence citation results specifically involve consistency verification between the policy, legal provisions, and voucher index entries cited in each hierarchical output of the student model and the citation entries corresponding to the hard labels of the sample hierarchical layers and the hierarchical citation entries output by the teacher model. The constraints on the final output results specifically involve precision alignment between the structured output content of each hierarchical layer of the student model and the hard labels of the sample hierarchical layers and the soft labels of the hierarchical layers output by the teacher model.

[0011] Furthermore, the edge-side adaptability verification process specifically involves deploying the trained student model to target edge devices with limited computing power. These edge devices include tax field service terminals, corporate financial audit terminals, and legal compliance verification terminals. A pre-annotated legal and tax audit stratified verification set is used to test three core indicators of the student model: single-sample inference latency, runtime memory usage, and stratified inference accuracy. When all three indicators reach the preset threshold, the student model is deemed to have passed verification.

[0012] Secondly, embodiments of this application provide a model hierarchical distillation and enhancement system for the vertical field of auditing, including: The sample preprocessing module is used to collect raw business data in the fields of legal and tax auditing. After cleaning, de-identifying, and standardizing the raw data, it constructs a hierarchical labeled training sample set and a hierarchical verification sample set according to the labeling requirements of five inference levels. The teacher model adaptation module is used to load a pre-trained large language model in the legal and tax audit domain as a teacher model, perform full hierarchical link inference on all samples in the training sample set, output the hierarchical soft label results corresponding to the five levels of each sample and store them in the training database. The expert scoring module is used to load the pre-trained expert scorer, score the quality of the output results of each level of the teacher model and the corresponding hierarchical hard label results of the samples, obtain the hierarchical weight coefficients of each of the five levels, and transmit them synchronously to the student model training module. The student model training module is used to build a lightweight student model with a hierarchical inference branch architecture. It iteratively trains the student model based on a hybrid constraint strategy and a distillation loss function, and monitors the loss reduction trend in real time during the training process. The edge verification module is used to deploy the initially trained student model to the target edge device, call the hierarchical verification sample set to perform hierarchical adaptation verification of the student model, and retrain the model after structural optimization for the model that fails the verification.

[0013] Thirdly, embodiments of this application provide an electronic device including a processor and a memory. The memory and the processor communicate with each other via an internal connection path. The memory stores instructions, and the processor executes the instructions stored in the memory. When the processor executes the instructions stored in the memory, it causes the processor to perform the method described in any of the above embodiments.

[0014] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, wherein when the computer program is run on a computer, the methods in any of the above-described embodiments are executed.

[0015] Compared with existing technologies, the beneficial effects of this invention are: This invention breaks down the model reasoning chain into five levels: fact identification, rule matching, element judgment, risk attribution, and disposal suggestions. It simultaneously applies hybrid constraints to three dimensions of the student model: intermediate reasoning representation, evidence citation results, and final output results. This solves the problems of intermediate reasoning logic deviation and non-compliant evidence citation caused by existing knowledge distillation schemes that only constrain the final output. As a result, the entire reasoning process of the student model meets the professional requirements of legal and tax auditing. The reasoning chain is traceable and verifiable, effectively meeting the core business requirements of legal and tax auditing for interpretability, compliance, and verifiability of model output.

[0016] This invention introduces a pre-trained expert scorer to quantify the importance of five inference levels. Based on the score results, corresponding level weight coefficients are assigned to different levels. This solves the problems of insufficient accuracy in core steps and redundant constraints in non-core steps caused by existing solutions that do not consider the differences in importance of different inference steps. The constraint strength of each level can be adjusted according to the actual needs of legal and tax audit business, which significantly improves the vertical scenario inference professionalism of the lightweight student model, while reducing the computational cost of model iteration training.

[0017] This invention solves the problem of large models being unable to be deployed on low-computing-power edge devices by adopting a student model architecture with a shared underlying feature extraction layer and layered independent upper-layer task heads, combined with an edge-side adaptability verification step. The trained student model has a low parameter count and can be adapted to various edge devices such as tax field service terminals, corporate financial audit terminals, and legal compliance verification terminals. It can complete the entire legal and tax audit inference without relying on cloud computing power support, thus lowering the computing power threshold for model deployment.

[0018] This invention is adaptable to various legal and tax audit business scenarios, such as tax risk verification, legal compliance review, and financial audit judgment. Model training and edge deployment can be completed without major architectural adjustments for different scenarios. This effectively reduces the R&D and iteration costs of lightweight models in the vertical field of legal and tax audit, improves the usability of lightweight models in front-line legal and tax audit business scenarios, and provides technical support for the digitalization and edge deployment of legal and tax audit business. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating the hierarchical distillation and enhancement method for models oriented towards the vertical audit field proposed in this invention. Figure 2 This is a schematic diagram of the architecture of the hierarchical distillation and enhancement system for the audit vertical field proposed in this invention. Figure 3 This is a structural block diagram of the electronic device proposed in this invention. Detailed Implementation

[0020] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of this application. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0021] Reference Figure 1 The first aspect of this application provides a hierarchical distillation and enhancement method for models in the vertical field of auditing, including: Step S101: Obtain a pre-trained large language model in the fields of law, taxation, and auditing as the teacher model, and obtain a lightweight small language model with fewer than a preset threshold of parameters as the student model to be trained.

[0022] The preset threshold is a critical value for the number of model parameters that is set in advance based on the computing power limit of target edge devices such as tax field service terminals, corporate financial audit terminals, and legal compliance verification terminals. Models with fewer parameters than this critical value have lower hardware operating costs and can meet the operating conditions of edge devices.

[0023] For example, when the target edge device is a portable terminal with 8G of memory, the preset threshold for the model parameter quantity is set to 7B parameters; if it is a lightweight terminal with 4G of memory, the preset threshold is set to 3B parameters. Those skilled in the art can flexibly adjust this threshold value according to the actual terminal hardware configuration.

[0024] The student model to be trained is a lightweight model that has not undergone initial training through stratified distillation for legal and tax audits. All stratified distillation and hybrid constraint training processes in this solution are performed on this student model.

[0025] For example, the pre-trained large language model in the fields of law, taxation and auditing can be the Llama and Qwen series large models after fine-tuning based on the professional corpus of finance, taxation and law; the lightweight student model uses the same series of small parameter base models, only adjusting the number of network parameters to adapt to the computing power of edge devices.

[0026] Step S102: Construct a hierarchically labeled training sample set for legal and tax audits. Each sample in the sample set corresponds to a labeling result at five levels: fact identification, rule matching, element judgment, risk attribution, and disposal suggestions.

[0027] During the construction of the sample set, the collected raw business data must first be cleaned, desensitized, and standardized in sequence. The cleaning operation includes filtering invalid blank documents, duplicate vouchers, and garbled text. The desensitization operation performs masking and replacement of sensitive fields such as company name, taxpayer identification number, and personal identity information. The standardization operation unifies the text encoding, document text format, and legal provision number format to ensure that the sample input format is consistent.

[0028] The layered annotation training sample set for legal and tax auditing is a dedicated training dataset generated according to a unified annotation quality control process. All five layers of annotation content have been cross-verified by senior experts in the field of legal and tax auditing, possessing high-standard professional quality control attributes.

[0029] Fact identification, rule matching, element judgment, risk attribution, and disposal recommendations are the five-layered reasoning levels that are fixed in this solution. The five levels are connected in sequence to form a complete closed-loop legal and tax audit reasoning chain, and each level corresponds to an independent audit business reasoning objective.

[0030] The five-layer annotation results corresponding to a single sample are hierarchical hard labels, which are standardized, professionally annotated data used for supervised training of student models.

[0031] Step S103: Call the teacher model to perform full-link inference on all samples in the training sample set, output the soft label results corresponding to the five levels of each sample, and call the pre-trained expert scorer to score the quality of the output results of each level of the teacher model and the hard label results corresponding to the stratification of the samples, so as to obtain the weight coefficients corresponding to each of the five levels.

[0032] The stratified soft label results are five-layer probabilistic reasoning results output by the teacher model's autonomous reasoning, which are different from manually labeled discrete stratified hard labels. Their function is to convey implicit auditing professional reasoning logic to the student model.

[0033] The pre-trained expert scorer is a professional quality assessment model that has been trained in advance using hierarchical labeled data from the legal and tax audit departments. It can quantitatively assess the degree of professional matching between the output results of a single level of teachers and the hierarchical hard labels.

[0034] The weight coefficients corresponding to the five levels are called level weight coefficients, which are used to allocate differentiated constraint strengths for different inference levels during the calculation of stratified distillation loss, and to adapt to the importance of the audit business corresponding to each level.

[0035] Step S104: The inference chain of the student model is divided into five levels of inference branches. Each inference branch receives the sample input features of the corresponding level and outputs the inference result of the corresponding level.

[0036] The student model's five-layer inference branch adopts a dedicated architecture with a shared bottom-level feature extraction layer and independent upper-level task heads. The bottom-level lightweight Transformer encoder uniformly completes the global text feature extraction work, while the upper layer sets up five mutually independent layered task heads.

[0037] The sample input features for the corresponding level are the standardized text hidden features output by the bottom feature extraction layer. Each inference branch only processes the audit inference task that matches its own level.

[0038] The inference results at the corresponding level include three types of data: hierarchical intermediate inference representation, evidence citation entries, and hierarchical structured output. These are also the constraint objects that are fully covered by the subsequent hybrid constraint strategies.

[0039] Step S105: Use a hybrid constraint strategy to iteratively train the student model. The hybrid constraint simultaneously covers the deviation constraints between the three types of objects in the student model—intermediate inference representations at each level, evidence citation results, and final output results—and the corresponding output results of the teacher model, as well as the hard label results of sample stratification.

[0040] The hybrid constraint strategy is a triple hierarchical joint constraint mechanism that executes three types of differentiated bias constraints simultaneously, with all three types of constraints participating in the calculation of the student model distillation loss in parallel.

[0041] Among them, the intermediate reasoning representation constraint is used to align the hidden layer output features of each hierarchical reasoning branch of the student model with the corresponding hierarchical hidden layer features of the teacher model; the evidence citation result constraint is used to verify the consistency between the policy, legal provisions, and voucher index entries of each hierarchical output of the student model and the hierarchical hard labels, and the hierarchical output citation entries of the teacher model; the final output result constraint is used to align the structured output content of each hierarchical output of the student model with the hierarchical hard labels, and the hierarchical soft labels of the teacher model.

[0042] Bias constraint refers to quantifying the error between the student model's output and the standard reference data using a loss function, and updating and optimizing all network parameters of the student model by relying on the error backpropagation mechanism.

[0043] During training, three independent loss values ​​are calculated: intermediate inference representation loss, evidence citation consistency loss, and hierarchical output precision loss. Then, the three types of losses corresponding to each layer are weighted and summed using the five-layer hierarchical weight coefficients obtained in step S103 to obtain the overall distillation total loss. All network parameters of the student model are updated based on the backpropagation of the total loss.

[0044] Step S106: Use the pre-annotated legal and tax audit validation set to perform edge-side adaptability verification on the trained student model, and output the enhanced student model that meets the edge-side deployment requirements.

[0045] The pre-annotated legal and tax audit validation set is a hierarchical validation sample set, which adopts the same five-layer hierarchical annotation specification as the training sample set. This dataset is independent of the training sample set and is only used for standardized testing of model performance and hardware compatibility metrics.

[0046] The edge-side adaptability verification is a standardized model validation process that uniformly tests three core performance indicators of student models: single-sample inference latency, memory usage, and hierarchical inference accuracy.

[0047] Meeting the requirements for edge deployment means that all three core performance indicators of the student model have reached their respective preset thresholds.

[0048] For example, the preset threshold for single-sample inference latency is 200ms, the preset threshold for runtime memory usage is 6GB, and the preset threshold for hierarchical inference accuracy is 92%. If all indicators meet the standards at the same time, the verification is deemed successful. The thresholds for each indicator can be flexibly adjusted according to the terminal's computing power and business accuracy requirements.

[0049] The enhanced student model is a lightweight audit inference model that has undergone hierarchical distillation, hybrid constraint training, and edge-side hierarchical adaptability verification. It can be directly deployed to various target edge devices and used in front-line legal and tax audit operations.

[0050] In one implementation, the total distillation loss during training is calculated using a hierarchical distillation loss function weighted by hierarchical expert scoring, the function expression of which is: ;in This represents the total distillation loss during student model training. This represents the weight coefficient obtained from the expert score corresponding to the i-th reasoning level, with a value ranging from 0 to 1, and is dimensionless. The predefined weighting coefficients for the intermediate inference loss are dimensionless. This represents the mean squared error loss of the intermediate inference representation of the student model at the i-th inference level, corresponding to the output of the teacher model. It is dimensionless. The predefined weighting coefficients representing the loss of evidence citation are dimensionless. This represents the cross-entropy loss between the evidence citation entries output by the student model at the i-th reasoning level and the corresponding outputs of the teacher model, as well as the hard-labeled citation entries for sample stratification. It is dimensionless. This represents the preset weighting coefficients for the final output loss; they are dimensionless. Let KL divergence loss be the KL divergence loss between the output of the student model at the i-th inference level and the hard labels of the sample stratification and the soft labels of each level of the teacher model. It is dimensionless.

[0051] Total distillation loss It is obtained by weighted summation of the component losses corresponding to the five reasoning levels, namely fact identification, rule matching, element judgment, risk attribution, and disposal suggestion, with corresponding subscripts. During training, backpropagation is performed based on the total distillation loss to synchronously update all network parameters of the student model.

[0052] The hierarchical weights are calculated by a pre-trained expert scorer after evaluating the output of the i-th level teacher model and the hard-labeled results of the sample stratification. Different weight coefficients can be assigned to different inference levels to match the differences in the importance of business at each audit inference level.

[0053] , , Preset weight coefficients for the whole system and configure them before model training starts. These coefficients are used to adjust the constraint strength among intermediate inference representation loss, evidence citation loss, and final output loss. Those skilled in the art can adjust the values ​​of the three sets of coefficients according to the needs of legal and tax audit business.

[0054] For the first The hierarchical intermediate inference representation loss uses mean squared error loss to align the hidden layer features of the hierarchical inference branches of the student model with the corresponding hierarchical hidden layer features of the teacher model, corresponding to the intermediate inference representation constraints within the hybrid constraint strategy.

[0055] For the first The layered evidence citation loss uses cross-entropy loss to quantify the evidence citation item matching deviation, thereby realizing the consistency verification of the evidence citation results of the student model and corresponding to the evidence citation result constraints within the hybrid constraint strategy.

[0056] For the first The final output loss for the hierarchical model is measured using KL divergence loss to measure the distribution difference between the hierarchical structured output and the hierarchical hard and soft labels, corresponding to the final output constraint within the hybrid constraint strategy.

[0057] Within each level, first go through , , The three independent loss categories are weighted and fused together, and then combined with the expert score weights corresponding to that level. The losses are weighted hierarchically; finally, the weighted losses of all five inference levels are summed to obtain the total distillation loss. .

[0058] In one implementation, the pre-trained expert scorer is constructed as follows: a hierarchical reasoning quality scoring dataset labeled by senior experts in the fields of law, taxation, and auditing is pre-collected. The dataset covers three core scenarios: tax risk verification, legal compliance review, and financial audit judgment. It includes three types of scoring results: accuracy, compliance, and relevance, corresponding to output samples at different reasoning levels. A lightweight text evaluation model is used to conduct supervised training on the scoring dataset. After training, the expert scorer automatically outputs a hierarchical weight coefficient in the range of 0 to 1 for any input hierarchical reasoning output result. The magnitude of the hierarchical weight coefficient is positively correlated with the quality of the corresponding hierarchical reasoning output result.

[0059] The samples in the hierarchical reasoning quality scoring dataset consist of reasoning text, evidence citation information, and structured output content from any level of the five-level reasoning hierarchy in legal, tax, and auditing. The sample annotations are the accuracy score, compliance score, and relevance score independently given by domain experts for the hierarchical reasoning result.

[0060] Accuracy score is used to assess whether the content of hierarchical reasoning matches the business facts and original voucher information; compliance score is used to assess whether the reasoning logic, legal provisions and policy citations comply with the professional standards of legal and tax auditing; relevance score is used to assess whether the content of the current level of reasoning is in line with the audit business objectives of this level.

[0061] During the training phase, the scores for accuracy, compliance, and relevance are combined to calculate a comprehensive quality score, which serves as a supervision label for model training. The comprehensive quality score is then normalized and mapped to a value range of 0 to 1.

[0062] The selected lightweight text classification model features a lightweight encoder architecture, which is suitable for subsequent offline batch inference needs. It can efficiently complete the quality evaluation of a large number of hierarchical inference results without generating excessive computing power.

[0063] When the expert scorer is trained and put into use, it receives the hierarchical inference output of the teacher model and the hard labels of the sample hierarchies as input, and outputs the hierarchical weight coefficients of the corresponding levels. The higher the quality of the inference result, the larger the value of the output hierarchical weight coefficient, and the stronger the constraint force corresponding to that level in the hierarchical distillation loss calculation, thereby realizing the dynamic allocation of hierarchical constraint weights based on the inference quality.

[0064] The hierarchical weight coefficients output by the expert scorer are directly used in the hierarchical distillation loss function of the aforementioned hierarchical expert scoring weighting, realizing differentiated loss constraints for different inference levels, with the technical links corresponding to each other.

[0065] In one implementation, the process of constructing a hierarchically labeled training sample set for legal and tax audits is as follows: original business documents, electronic documents, financial vouchers, and field business verification records from three scenarios—tax risk verification, legal compliance review, and financial audit judgment—are collected as the original sample inputs for hierarchical labeling. For each sample input, the following are labeled sequentially: entity and event extraction results at the fact identification level; matching legal provisions and policy entries at the rule matching level; compliance judgment results at the requirement judgment level; risk point location and cause analysis results at the risk attribution level; and rectification and response plan results at the disposal suggestion level.

[0066] The original sample input for hierarchical annotation consists of first-hand business data that has not undergone reasoning or processing. It is the basic data source for legal, tax, and audit reasoning. All original sample inputs undergo cleaning, desensitization, and standardization before annotation to filter invalid data, mask various sensitive information, and unify text format to ensure that the samples are standardized and usable.

[0067] The five-level labeling strictly follows the audit reasoning sequence from basic information extraction to final solution output. The levels are interdependent and logically progressive, corresponding one-to-one with the five-level reasoning chain of fact identification, rule matching, element judgment, risk attribution, and disposal recommendations.

[0068] The entity and event extraction results at the fact recognition level refer to the extraction of objective business information such as transaction entities, fund flows, business behaviors, and time nodes from the original sample input. It only objectively restores the business facts and does not conduct compliance assessments.

[0069] The matching results of the rule matching level refer to the retrieval and matching of applicable legal provisions and policy entries based on the identified objective business facts, thereby establishing the relationship between business facts and regulatory basis.

[0070] The compliance determination result at the requirement judgment level refers to the verification of whether the business meets the statutory requirements by combining the completed and matched normative items, and outputting three types of judgment conclusions: business compliance, business non-compliance, and compliance defects.

[0071] The risk attribution level identifies risk points and analyzes causes, referring to the specific risk locations and identifies the business sources and underlying causes of risks for business content that is determined to be non-compliant or have compliance flaws.

[0072] The results of the rectification and response plan at the disposal recommendation level refer to the output of implementable rectification measures, risk prevention plans, and compliance optimization suggestions based on risk positioning and causal analysis.

[0073] After a single original sample input completes five layers of annotation, a complete set of layered annotation data is formed, which serves as the layered hard label. The collection of all annotated samples constitutes the layered annotation training sample set for legal and tax auditing, which is used for subsequent teacher model inference and student model supervised training.

[0074] In one implementation, the five-level inference branches of the student model are built using an architecture that shares a bottom-level feature extraction layer and consists of independent upper-level task heads. The bottom-level feature extraction layer is completed using a lightweight Transformer encoder. The number of attention heads and the dimension of the hidden layers in the encoder are preset according to the computing power threshold of the target device. Each independent task head is set with an adapted output layer structure for the task type of the corresponding level. The fact recognition level task head is set with a sequence labeling output structure, the rule matching level task head is set with a multi-label classification output structure, the element judgment level task head is set with a binary classification output structure, the risk attribution level task head is set with a controllable text generation output structure, and the disposal suggestion level task head is set with a controllable text generation output structure.

[0075] Among them, the shared underlying feature extraction layer is used to uniformly encode the features of the original sample text, extract the globally common text hidden features from the input text, and all five inference branches share the feature information output by the encoder, avoiding repeated feature calculations, reducing the overall computing power consumption of the model, and adapting to the lightweight deployment requirements of the edge.

[0076] The number of attention heads and the dimension of hidden layers in a lightweight Transformer encoder directly determine the number of encoder parameters and inference overhead. By presetting parameters in conjunction with the computing power threshold of the target edge device, a balance can be achieved between model expressive power and inference latency and memory usage. Those skilled in the art can flexibly adjust the above parameters according to hardware conditions.

[0077] The hierarchical independent upper-level task heads are independent of each other, receiving unified features output from the lower-level feature extraction layer, and each completing its own exclusive inference task, thus achieving hierarchical inference decoupling and facilitating hierarchical constraint training and hierarchical effect evaluation.

[0078] The fact recognition hierarchical configuration sequence labeling output structure is adapted to entity and event element extraction tasks, and can locate the corresponding business entities, behaviors, time and other objective fact elements in the text.

[0079] The rule matching hierarchy is configured with a multi-label classification output structure, which is suitable for scenarios that match multiple legal regulations, policies, and auditing standards at the same time, and supports the output of multiple independent adaptive standard entries.

[0080] The requirement judgment hierarchy is configured with a binary output structure to output the judgment result of whether the business matter meets the compliance requirements, thereby realizing the binary judgment of compliance.

[0081] The risk attribution hierarchy is configured with a controllable text generation output structure, which outputs descriptions of risk location, risk causes, etc. in structured text form. The generated content is constrained by the reasoning results of the preceding hierarchy, ensuring that the reasoning content is traceable.

[0082] The hierarchical configuration of disposal recommendations allows for the generation of controllable text output structures. Based on the identified risk information, targeted rectification plans and compliance response suggestions are generated. The generation process is constrained by the results of prior hierarchical reasoning, ensuring that the output content meets the needs of the audit scenario.

[0083] The standardized text hidden features output by the bottom feature extraction layer are used as input to the independent task heads of each layer. This corresponds to the technical solution mentioned above where each inference branch receives the input features of the corresponding layer's samples. The architecture design is well-matched with the overall training process.

[0084] In one implementation, during the execution of the hybrid constraint strategy, the constraints on the intermediate inference representation specifically involve feature alignment between the output of the last hidden layer of each hierarchical inference branch of the student model and the hidden layer output of the corresponding hierarchical inference link of the teacher model. The constraints on the evidence citation results specifically involve consistency verification between the policy, legal provisions, and voucher index entries cited in each hierarchical output of the student model and the citation entries corresponding to the hard labels of the sample hierarchical layers and the hierarchical citation entries output by the teacher model. The constraints on the final output results specifically involve precision alignment between the structured output content of each hierarchical layer of the student model and the hard labels of the sample hierarchical layers and the soft labels of the hierarchical layers output by the teacher model.

[0085] In this context, intermediate inference represents the constraint applied to the hidden features during the model's inference process, independent of the final text output. Its purpose is to allow the student model to learn the teacher model's inherent inference logic and feature distribution, rather than simply replicating the final output text. The loss corresponding to this constraint is the loss function described in the previous section on hierarchical distillation. .

[0086] Feature alignment quantifies the difference between the hidden layer feature vectors of the student model and the corresponding hidden layer feature vectors of the teacher model using a loss function, and adjusts the parameters of the student model based on backpropagation to reduce the feature distribution gap between the two. The feature alignment operation is performed independently at each inference level, and there is no confusion between levels.

[0087] The evidence citation constraint focuses on the source information of the evidence used in the audit reasoning process, ensuring the accuracy of the policies, laws, and business documents cited in the student model's reasoning, and avoiding problems such as unfounded reasoning and incorrect citation of norms; the loss corresponding to this constraint is the loss function in the stratified distillation mentioned above. .

[0088] The consistency verification references two types of standards: one is the standard reference entries corresponding to the hard labels of the sample layers obtained by manual annotation, and the other is the layered reference entries output by the teacher model inference. It simultaneously integrates human professional knowledge and the reasoning experience learned by the teacher model.

[0089] The final output constraint is directed towards the structured reasoning conclusions output by the model, ensuring the accuracy of the student model's output conclusions at each layer and simultaneously fitting manually labeled hard labels and teacher-generated layered soft labels. The loss corresponding to this constraint is the layered distillation loss function mentioned earlier. .

[0090] Precision alignment relies on a loss function to measure the difference between the hierarchical output distribution of the student model and the standard reference data distribution, thereby achieving supervised optimization at the output conclusion level. Precision alignment and intermediate layer feature alignment work together to constrain the training of the student model from two dimensions: internal reasoning logic and external output conclusion.

[0091] The three types of constraints are executed synchronously and in parallel, and jointly participate in the calculation of the total distillation loss. The three types of constraints correspond to different stages of the hierarchical reasoning link of the student model, forming a full-link constraint system from hidden features and reasoning basis to the final conclusion, constituting a complete hybrid constraint strategy.

[0092] All constraints are executed according to the principle of one-to-one matching at each level. The fact recognition level only performs constraint calculations with the output of the fact recognition level of the teacher model, the hard label and soft label of that level, and does not perform cross-level matching. It strictly matches the five-layer hierarchical reasoning architecture.

[0093] In one implementation, the edge-side adaptability verification process specifically involves deploying the trained student model to a target edge device with limited computing power. The edge device includes a tax field service terminal, a corporate financial audit terminal, and a legal compliance verification terminal. A pre-annotated legal and tax audit stratified verification set is used to test three core indicators of the student model: single-sample inference latency, runtime memory usage, and stratified inference accuracy. When all three indicators reach a preset threshold, the student model is deemed to have passed the verification.

[0094] The target edge device is a local offline device. Compared with cloud servers, the device's computing power and memory resources are significantly limited. The edge-side compatibility verification is used to check whether the student model can run stably on local hardware and meet the business accuracy requirements without the cloud environment.

[0095] The legal and tax audit stratified validation set is independent of the training sample set and does not participate in the student model training process. It adopts the same five-layer stratified annotation standard as the training sample set to avoid data leakage that could lead to distorted test results and ensure that the validation results are objective and reliable.

[0096] The single-sample inference latency metric measures the total computation time required for the model to process a single legal and tax audit sample, reflecting the terminal business response speed; the runtime memory usage metric measures the amount of device memory resources used during model loading and inference, avoiding device memory overflow and program crashes.

[0097] The hierarchical reasoning accuracy index measures the reasoning accuracy of each of the five reasoning levels: fact identification, rule matching, element judgment, risk attribution, and disposal suggestions. It comprehensively measures the hierarchical reasoning ability of the model, which is different from the global single output accuracy and can accurately locate the performance shortcomings of each reasoning level.

[0098] The preset thresholds for the three types of indicators can be pre-set based on the hardware specifications of edge devices and the practical requirements of front-line legal and tax audit business. Only when the three indicators of latency, memory, and accuracy simultaneously meet the threshold requirements can the model be considered to balance operating efficiency and inference accuracy and be ready for deployment.

[0099] If any core indicator fails to reach the preset threshold, the verification is deemed unsuccessful. The number of student model parameters, network structure, or hierarchical distillation training needs to be readjusted until all indicators meet the standards and an enhanced student model that can be deployed is obtained.

[0100] like Figure 2 As shown, a second aspect of this application provides a hierarchical model distillation and enhancement system 200 for the audit vertical domain, used to execute the hierarchical model distillation and enhancement method for the audit vertical domain according to any of the embodiments of the first aspect described above. The hierarchical model distillation and enhancement system 200 for the audit vertical domain includes a sample preprocessing module 210, a teacher model adaptation module 220, an expert scoring module 230, a student model training module 240, and an edge verification module 250.

[0101] Specifically, the sample preprocessing module 210 collects raw business data from the legal and tax auditing domain. After cleaning, de-identifying, and standardizing the raw data, it constructs a hierarchical labeled training sample set and a hierarchical validation sample set according to the labeling requirements of five inference levels. The teacher model adaptation module 220 loads a pre-trained large language model from the legal and tax auditing domain as the teacher model. It performs full hierarchical link inference on all samples in the training sample set, outputs the hierarchical soft label results corresponding to the five levels for each sample, and stores them in the training database. The expert scoring module 230 loads a pre-trained expert scorer to score the quality of the output results of each level of the teacher model and the hierarchical hard label results corresponding to the samples. It obtains the hierarchical weight coefficients corresponding to each of the five levels and transmits them synchronously to the student model training module 240. The student model training module 240 builds a lightweight student model with a hierarchical inference branch architecture. It iteratively trains the student model based on a hybrid constraint strategy and a distillation loss function, and monitors the loss reduction trend in real time during the training process. The edge verification module 250 is used to deploy the initially trained student model to the target edge device, call the hierarchical verification sample set to perform hierarchical adaptation verification on the student model, and retrain the model after structural optimization for the model that fails the verification.

[0102] The various modules within the system work together and flow sequentially to complete the complete model stratified distillation and enhancement process; the modules interact with each other through the training database, and the data transmitted between the modules includes structured information such as stratified hard labels, stratified soft labels, stratified weight coefficients, and sample set data.

[0103] The hierarchical labeled training sample set and hierarchical verification sample set output by the sample preprocessing module 210 are independent of each other. The hierarchical verification sample set is only used for the final model performance verification and does not participate in the model training process, thus avoiding data leakage that could cause the verification results to be distorted. The five inference levels are, in order, fact recognition, rule matching, element judgment, risk attribution, and disposal suggestions.

[0104] The teacher model adaptation module 220 uses a professional large language model to carry out hierarchical reasoning and generate hierarchical soft labels. The hierarchical soft labels and manually labeled hierarchical hard labels together constitute the dual supervision basis for student model training, providing teacher model reasoning knowledge for hierarchical distillation training.

[0105] The hierarchical weight coefficients output by the expert scoring module 230 are then input into the student model training module 240 and used as weight parameters in the hierarchical distillation loss function. This enables dynamic adjustment of the constraint ratio of different reasoning levels in the total distillation loss based on the reasoning quality of each level, and connects the expert scoring link with the model training loss calculation link.

[0106] The lightweight student model built by the student model training module 240 adopts a hierarchical inference branch architecture with a shared bottom-level feature extraction layer and hierarchical independent upper-level task heads. During the training process, a hybrid constraint strategy consisting of three types of constraints—intermediate inference representation, evidence citation results, and final output results—is simultaneously enabled, and the hierarchical distillation loss function with hierarchical expert scoring weights is used to complete the iterative update of model parameters.

[0107] The training process monitors the loss reduction trend in real time. If the total distillation loss no longer decreases significantly after multiple iterations, the model is determined to have converged, the iterative training is stopped, and the initially trained student model is output.

[0108] The edge verification module 250 performs hierarchical adaptability verification, corresponding to the edge adaptability verification process described above. It evaluates the model's ability to be deployed on edge devices from multiple dimensions, including inference latency, memory usage, and hierarchical inference accuracy. For student models that fail verification, the Transformer encoder parameters, task head structure, and model parameter quantity can be adjusted to complete structural optimization, and then sent back to the student model training module 240 for retraining.

[0109] The system as a whole forms a closed-loop workflow of "sample processing → teacher model knowledge generation → expert dynamic weight calculation → hierarchical distillation training → edge-side implementation and verification iteration", which fully realizes the distillation enhancement and edge-side adaptation of lightweight student models in legal and tax audit scenarios.

[0110] The following two examples further illustrate this point: This invention discloses a hierarchical distillation and enhancement method for models in the vertical field of auditing. First, a pre-trained large language model in the legal and tax auditing domain is acquired as the teacher model. Then, a lightweight small language model with fewer than a preset threshold of parameters is acquired as the student model to be trained. The teacher model is pre-trained continuously on millions of publicly available legal and tax audit documents, policy provisions, and audit cases, with a parameter count set to the tens of billions level. It only supports cloud deployment and operation, and its inference accuracy is close to that of senior legal and tax audit professionals. The student model to be trained has fewer than one billion parameters, adapting to the computing power limit of edge devices, and can complete inference operations offline.

[0111] Next, a hierarchically labeled training sample set for legal and tax auditing is constructed. Each sample in the set corresponds to a labeling result at five levels: fact identification, rule matching, element judgment, risk attribution, and disposal suggestion. During the sample collection process, the raw data is first cleaned, de-identified, and standardized to remove all information involving personal privacy and corporate trade secrets. The text format and coding standards are unified. Then, qualified professionals complete the hierarchical labeling. All labeling results are cross-verified by at least two professionals before being officially stored in the database to ensure that the labeling accuracy of the training samples meets the professional requirements of legal and tax auditing, forming standardized hierarchical hard labels.

[0112] The teacher model performs end-to-end inference on all samples in the training sample set, outputting stratified soft labels for each sample across five levels. Simultaneously, a pre-trained expert scorer evaluates the quality of the teacher model's output at each level and the corresponding stratified hard labels, yielding the level weight coefficients for each of the five levels. The expert scorer is a lightweight text evaluation model, pre-trained in supervised mode on hundreds of thousands of quality samples labeled by senior legal, tax, and auditing experts. It automatically assigns a level weight coefficient ranging from 0 to 1 to any input level reasoning output; a higher weight coefficient indicates a higher level of professionalism in the reasoning result and provides stronger guidance for student model training.

[0113] The student model's inference chain is divided into five levels of inference branches. Each inference branch receives sample input features from its corresponding level and outputs the inference result for that level. The student model adopts an architecture with a shared bottom-level feature extraction layer and independent upper-level task heads. The bottom-level feature extraction layer is implemented using a lightweight Transformer encoder, with the number of attention heads and the dimension of hidden layers pre-set according to the computing power threshold of the target device. Each of the five levels has an independent task head: the fact recognition level task head uses a sequence labeling output structure; the rule matching level task head uses a multi-label classification output structure; the requirement judgment level task head uses a binary classification output structure; the risk attribution level task head uses a controllable text generation output structure; and the disposal suggestion level task head uses a controllable text generation output structure. Each task head only adapts its output for the corresponding level's task and does not share upper-level parameters, thus avoiding inference interference between different levels of tasks.

[0114] A hybrid constraint strategy is employed for iterative training of the student model. This hybrid constraint simultaneously covers deviation constraints between the student model's intermediate inference representations at each level, evidence citation results, and final output results, and the corresponding output results of the teacher model, as well as the hard-labeled results of the sample stratification. The intermediate inference representation constraint aligns the output of the last hidden layer of each inference branch in the student model with the hidden layer output of the corresponding inference step in the teacher model, ensuring consistency between the student model's inference representation logic and the teacher model's. The evidence citation constraint verifies the consistency between the policy, legal, and voucher index entries output by the student model at each level and the citation entries labeled in the samples and output by the teacher model, ensuring the compliance and accuracy of evidence citation. The final output constraint aligns the structured output content of each level of the student model with the hard labels of the sample stratification and the soft labels of the teacher model, ensuring the professionalism of the output results. During training, the total distillation loss is calculated by combining the weight coefficients of each level, performing a weighted summation of the loss values ​​corresponding to the three types of constraints, and dynamically adjusting the constraint strength of each level according to the weights to ensure the training accuracy of the core inference levels.

[0115] The trained student model is validated for edge-side adaptability using a pre-labeled legal and tax audit validation set, outputting an enhanced student model that meets edge deployment requirements. During validation, three core metrics are tested: single-sample inference latency, runtime memory usage, and hierarchical inference accuracy. The student model is considered validated if all three metrics reach preset thresholds. If any metric fails to reach a threshold, the student model's task header structure and hidden layer dimensions are pruned and optimized before retraining until validation is successful.

[0116] The hierarchical distillation and enhancement system 200 for auditing vertical fields, which is the basis of this method, includes a sample preprocessing module 210, a teacher model adaptation module 220, an expert scoring module 230, a student model training module 240, and an end-side verification module 250. The modules interact with each other through standardized interfaces. All data transmission processes are encrypted to ensure the security of legal and tax audit data. The trained model can be stored in a general computer-readable storage medium or integrated into various electronic devices.

[0117] The first embodiment is applied to the deployment scenario of tax risk verification on the terminal side of grassroots tax authorities.

[0118] The training samples collected by the sample preprocessing module 210 consist entirely of VAT declaration data, invoice data, tax audit cases, and currently valid tax policy provisions from the past three years within the jurisdiction. After the raw data is collected, it undergoes full cleaning, desensitization, and standardization to remove sensitive information such as the contact information of the legal representative of the enterprise and the core operating data of the enterprise. The formats of various business texts are standardized. Then, staff with more than three years of tax audit experience complete the hierarchical labeling. During the labeling process, the focus is on improving the labeling accuracy of the risk attribution level. The labeling results of all samples are cross-verified by two or more professionals before being included in the training sample set to form standardized hierarchical hard-labeled sample data.

[0119] The teacher model adaptation module 220 utilizes a teacher model that has been pre-tuned specifically for the tax risk verification scenario. Its accuracy in identifying tax risk points meets the requirements of tax business specifications. Throughout the entire reasoning process, it synchronously outputs intermediate representations, evidence citations, and reasoning explanations at each level. The generated hierarchical soft labels can be directly used for student model training. The expert scoring module 230 assigns corresponding hierarchical weight coefficients to the five reasoning levels based on the business needs of the tax risk verification scenario. The weight for the fact identification level is set to 0.15, the rule matching level to 0.2, the element judgment level to 0.2, the risk attribution level to 0.3, and the disposal suggestion level to 0.15. The risk attribution level has the highest weight, aligning with the core requirement of prioritizing the identification of risk causes in tax risk verification.

[0120] The student model training module 240 sets the number of attention heads in the underlying encoder of the student model to be adapted to the computing power requirements of tax field service terminals. The preset threshold for inference latency is within 2 seconds per sample, and the threshold for memory usage is within 4GB. During training, a hybrid constraint strategy is used to focus on verifying the evidence citation process, requiring that the document number, effective date, and scope of application of all cited tax policy provisions be completely consistent with the officially released content to avoid policy citation errors. The decreasing trend of the total loss is monitored in real time during training. When the loss value no longer decreases for ten consecutive training epochs, the model is considered to have converged, and training is stopped to obtain the initial student model.

[0121] The edge verification module 250 deploys the trained student model to the tax field service terminal, using two thousand pre-labeled tax risk verification samples for edge compatibility verification. During testing, the input scenario of on-site field verification is simulated. Input data includes compiled enterprise declaration data, on-site voucher recognition results, and compiled enterprise inquiry records. All input data is standardized before being input into the student model. The student model outputs end-to-end hierarchical inference results, allowing business personnel to directly view the inference process and evidence citations at each level to verify the compliance of the inference results. Once all test indicators reach preset thresholds, the model is deployed in batches to all tax field service terminals within the jurisdiction.

[0122] This embodiment employs a stratified distillation training method, enabling a lightweight student model to be directly deployed and run on tax field service terminals. It can complete the entire tax risk verification and reasoning process without relying on cloud computing power, solving the problem that existing large models cannot run offline at field operation sites. During training, constraints are simultaneously applied to intermediate reasoning representations, evidence citations, and final outputs, ensuring that the student model's reasoning logic meets the professional requirements of tax verification. The reasoning chain is traceable and verifiable, satisfying the compliance requirements of tax operations. Higher constraint weights are set for the risk attribution level, improving the accuracy of core reasoning stages. This makes the risk identification results of the student model approach the level of large models, effectively reducing the workload of frontline field service personnel and improving the efficiency of tax risk verification.

[0123] The second embodiment is applied to the routine self-inspection scenario of legal and tax audits within enterprises.

[0124] The training samples collected by the sample preprocessing module 210 consist of the company's financial vouchers, audit reports, compliance review records, applicable civil and commercial law provisions, tax policy provisions, and internal compliance management system documents from the past five years. After the raw data is collected, it undergoes full cleaning, desensitization, and standardization to remove sensitive content such as core technical information and core customer information, and to unify the document and voucher text formats. Then, the company's internal audit department and the staff of the external law firm jointly complete the hierarchical annotation. During the annotation process, the focus is on improving the annotation accuracy of rule matching and requirement judgment levels. All annotation results are cross-verified by the staff of both parties and then included in the training sample set to form standardized hierarchical hard-labeled sample data.

[0125] The teacher model adaptation module 220 utilizes a teacher model that has been pre-tuned to meet the legal and tax audit requirements of the company's industry. Its accuracy in identifying common industry compliance risk points meets the requirements of both internal auditing and industry regulation. Throughout the entire reasoning process, it synchronously outputs evidence citations and reasoning logic explanations at each level, and the generated layered soft labels can be directly used for student model training. The expert scoring module 230, designed for the company's routine compliance self-inspection needs, assigns corresponding weight coefficients to the five reasoning levels: fact identification (0.1), rule matching (0.25), requirement judgment (0.3), risk attribution (0.2), and action recommendation (0.15). The rule matching and requirement judgment levels have the highest weights, aligning with the company's core requirement of prioritizing compliance assessment during routine compliance self-inspections.

[0126] The student model training module 240 sets the number of attention heads in the underlying encoder of the student model to be adapted to the computing power requirements of the office terminals used by the enterprise's finance department. The preset threshold for inference latency is within 3 seconds per sample, and the threshold for memory usage is within 6GB. During training, a hybrid constraint strategy is used to focus on verifying the evidence citation process, requiring that all cited internal enterprise rules and regulations, industry regulatory policies, applicable scenarios, and effective dates be completely consistent with the officially released content to avoid errors in rule application. The decreasing trend of the total loss is monitored in real time during training. When the loss value no longer decreases for ten consecutive training epochs, the model is considered to have converged, and training is stopped to obtain the initial student model.

[0127] The edge verification module 250 deploys the trained student model to the enterprise's internal office terminals, using 3,000 pre-labeled internal audit verification samples for edge compatibility verification. During testing, it simulates the input scenarios of daily self-checks by finance personnel. Input data includes monthly financial statements, expense vouchers, contract text recognition results, employee travel records, etc. All input data is standardized before being input into the student model. The student model outputs end-to-end hierarchical inference results, allowing finance personnel to directly view the inference process at each level, quickly locate compliance risk points, and generate self-check reports. Once all test indicators reach preset thresholds, the model is deployed to the office terminals of all finance personnel within the enterprise.

[0128] This embodiment employs a stratified distillation training method, enabling the lightweight student model to be deployed and run directly on enterprise internal office terminals. All audit data is processed locally within the enterprise, eliminating the need to upload to the cloud and avoiding the risk of leakage of sensitive financial data. Constraints are applied to the entire inference process during training to ensure that the student model's inference logic complies with enterprise compliance and industry regulatory requirements. The inference process is interpretable and verifiable, meeting the compliance requirements of internal audits. Higher constraint weights are set for rule matching and requirement judgment levels, improving the accuracy of core compliance review processes. This allows the self-inspection results of the student model to approach the professionalism of larger models, effectively reducing the manpower costs of internal audits and improving the coverage and efficiency of routine compliance self-inspections.

[0129] Figure 3 A structural block diagram of an electronic device according to an embodiment of the present invention is shown. Figure 3 As shown, the electronic device includes a memory 310 and a processor 320. The memory 310 stores a computer program that can run on the processor 320. When the processor 320 executes the computer program, it implements the method described in the above embodiments. The number of memories 310 and processors 320 can be one or more.

[0130] The electronic device also includes: The communication interface 330 is used to communicate with external devices and perform data exchange and transmission.

[0131] If the memory 310, processor 320, and communication interface 330 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 3 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0132] Optionally, in a specific implementation, if the memory 310, processor 320 and communication interface 330 are integrated on a single chip, the memory 310, processor 320 and communication interface 330 can communicate with each other through an internal interface.

[0133] This invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method provided in this application.

[0134] This application also provides a chip, which includes a processor for calling and executing instructions stored in a memory, causing a communication device on which the chip is installed to perform the method provided in this application.

[0135] This application also provides a chip, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute the method provided in the application embodiment.

[0136] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting the Advanced Reduced Instruction Set Computing (RISC) machine (ARM) architecture.

[0137] Further, optionally, the aforementioned memory may include read-only memory and random access memory, and may also include non-volatile random access memory. The memory may be volatile or non-volatile, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. Many forms of RAM are available by way of example, but not limitation. Examples include static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0138] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0139] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0140] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0141] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process. Furthermore, the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functionality involved.

[0142] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0143] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware, the program being stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiments.

[0144] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0145] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A hierarchical distillation and enhancement method for models in the vertical field of auditing, characterized in that, include: We acquire pre-trained large language models in the fields of law, taxation, and auditing as teacher models, and lightweight small language models with fewer than a preset threshold of parameters as student models to be trained. Construct a hierarchical labeled training sample set for legal and tax auditing. Each sample in the sample set corresponds to a labeling result at five levels: fact identification, rule matching, element judgment, risk attribution, and disposal suggestion. The teacher model is invoked to perform full-link inference on all samples in the training sample set, and outputs the soft label results corresponding to the five levels of each sample. At the same time, the pre-trained expert scorer is invoked to score the quality of the output results of the teacher model at each level and the hard label results corresponding to the stratification of the samples, so as to obtain the weight coefficients corresponding to each of the five levels. The inference chain of the student model is divided into five levels of inference branches. Each inference branch receives the sample input features of the corresponding level and outputs the inference result of the corresponding level. A hybrid constraint strategy is used to iteratively train the student model. The hybrid constraint simultaneously covers the deviation constraints between the three types of objects in the student model—intermediate inference representations at each level of the student model, evidence citation results, and final output results—and the corresponding output results of the teacher model, as well as the hard label results of sample stratification. The trained student model is validated for edge-side adaptability using a pre-labeled legal and tax audit validation set, and an enhanced student model that meets edge-side deployment requirements is output.

2. The model hierarchical distillation and enhancement method for vertical audit domains according to claim 1, characterized in that, The total distillation loss during training is calculated using a stratified distillation loss function weighted by stratified expert scores. The function expression is as follows: ;in This represents the total distillation loss during student model training. This represents the weighting coefficient obtained from the expert score corresponding to the i-th reasoning level. This represents the preset weighting coefficients for the intermediate inference loss. Let represent the mean squared error loss of the intermediate inference representation of the student model at the i-th inference level compared to the corresponding output of the teacher model. The pre-defined weighting coefficients representing the loss of evidence citation. Let represent the cross-entropy loss between the evidence citation entries output by the student model at the i-th reasoning level and the corresponding outputs of the teacher model, as well as the hard-labeled citation entries for sample stratification. This represents the preset weighting coefficients for the final output loss. This represents the KL divergence loss between the output of the student model at the i-th inference level and the hard labels of the sample hierarchy and the soft labels of each level of the teacher model.

3. The model hierarchical distillation and enhancement method for vertical audit domains according to claim 2, characterized in that, The construction process of the pre-trained expert scorer is as follows: a hierarchical reasoning quality scoring dataset labeled by senior experts in the fields of law, taxation and auditing is collected in advance. The dataset covers three core scenarios: tax risk verification, legal compliance review, and financial audit judgment. It contains three types of scoring results: accuracy, compliance, and relevance, corresponding to output samples of different reasoning levels. A lightweight text evaluation model is used to conduct supervised training on the scoring dataset. After training, the expert scorer automatically outputs a hierarchical weight coefficient in the range of 0 to 1 for any input hierarchical reasoning output result. The value of the hierarchical weight coefficient is positively correlated with the quality of the corresponding hierarchical reasoning output result.

4. The model hierarchical distillation and enhancement method for vertical audit domains according to claim 1, characterized in that, The process of constructing a hierarchically labeled training sample set for legal and tax auditing is as follows: original business documents, electronic documents, financial vouchers, and field business verification records from three scenarios—tax risk verification, legal compliance review, and financial audit judgment—are collected as the original sample inputs for hierarchical labeling. For each sample input, the following are labeled sequentially: entity and event extraction results at the fact identification level; matching legal provisions and policy entries at the rule matching level; compliance judgment results at the requirement judgment level; risk point location and cause analysis results at the risk attribution level; and rectification and response plan results at the disposal suggestion level.

5. The model hierarchical distillation and enhancement method for vertical audit domains according to claim 1, characterized in that, The student model's five-level inference branches are built using a shared bottom-level feature extraction layer and a hierarchical, independent upper-level task head architecture. The bottom-level feature extraction layer is implemented using a lightweight Transformer encoder. The number of attention heads and the dimension of the hidden layers in the encoder are preset according to the computing power threshold of the target device. Each hierarchical, independent task head sets an adapted output layer structure for the corresponding task type at the corresponding level. The fact recognition level task head is set to a sequence labeling output structure, the rule matching level task head is set to a multi-label classification output structure, the element judgment level task head is set to a binary classification output structure, the risk attribution level task head is set to a controllable text generation output structure, and the disposal suggestion level task head is set to a controllable text generation output structure.

6. The model hierarchical distillation and enhancement method for audit vertical domains according to claim 1, characterized in that, During the execution of the hybrid constraint strategy, the constraints on the intermediate inference representation are specifically to align the output of the last hidden layer of each hierarchical inference branch of the student model with the hidden layer output of the corresponding hierarchical inference link of the teacher model. The constraints on the evidence citation results are specifically to perform consistency verification on the policy, legal provisions, and voucher index entries cited in each hierarchical output of the student model with the citation entries corresponding to the hard labels of the sample hierarchical layer and the hierarchical citation entries output by the teacher model. The constraints on the final output results are specifically to align the structured output content of each hierarchical layer of the student model with the hard labels of the sample hierarchical layer and the soft labels of the hierarchical layer output by the teacher model.

7. The hierarchical distillation and enhancement method for models oriented towards the vertical audit domain according to claim 1, characterized in that, The edge-side adaptability verification process involves deploying the trained student model to target edge devices with limited computing power. These edge devices include tax field service terminals, corporate financial audit terminals, and legal compliance verification terminals. A pre-annotated legal and tax audit stratified verification set is used to test three core indicators of the student model: single-sample inference latency, runtime memory usage, and stratified inference accuracy. When all three indicators reach the preset threshold, the student model is deemed to have passed the verification.

8. A model hierarchical distillation and enhancement system for the audit vertical domain, used to execute the model hierarchical distillation and enhancement method for the audit vertical domain as described in any one of claims 1 to 7, characterized in that, include: The sample preprocessing module is used to collect raw business data in the fields of legal and tax auditing. After cleaning, de-identifying, and standardizing the raw data, it constructs a hierarchical labeled training sample set and a hierarchical verification sample set according to the labeling requirements of five inference levels. The teacher model adaptation module is used to load a pre-trained large language model in the legal and tax audit domain as a teacher model, perform full hierarchical link inference on all samples in the training sample set, output the hierarchical soft label results corresponding to the five levels of each sample and store them in the training database. The expert scoring module is used to load the pre-trained expert scorer, score the quality of the output results of each level of the teacher model and the corresponding hierarchical hard label results of the samples, obtain the hierarchical weight coefficients of each of the five levels, and transmit them synchronously to the student model training module. The student model training module is used to build a lightweight student model with a hierarchical inference branch architecture. It iteratively trains the student model based on a hybrid constraint strategy and a distillation loss function, and monitors the loss reduction trend in real time during the training process. The edge verification module is used to deploy the initially trained student model to the target edge device, call the hierarchical verification sample set to perform hierarchical adaptation verification of the student model, and retrain the model after structural optimization for the model that fails the verification.

9. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores instructions that are loaded and executed by the processor to implement the model hierarchical distillation and enhancement method for audit vertical domains as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the model hierarchical distillation and enhancement method for audit vertical domains as described in any one of claims 1 to 7.

Images