Training method of case factor extraction model and related device
By iteratively training the case element extraction model from judicial documents and utilizing the evaluation metrics of labeled datasets and multi-label classification tasks, the problem of low efficiency in identifying key case elements in judicial documents has been solved, achieving higher accuracy and comprehensiveness, and making it suitable for applications in multiple fields.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-04
- Publication Date
- 2026-03-31
AI Technical Summary
In existing technologies, the identification efficiency of key case elements in judicial documents is low, and the classification accuracy is insufficient, failing to meet the comprehensiveness and accuracy requirements of automated extraction.
By constructing a case element extraction model, iteratively training the pre-trained initial model using labeled datasets, and adjusting the model using evaluation metrics from multi-label classification tasks, such as global recall, global precision, and F1 score, to achieve the preset benchmark evaluation metrics, thereby improving the model's accuracy and comprehensiveness.
It improved the accuracy and comprehensiveness of case element extraction, enhanced the model's adaptability across multiple domains, and optimized the model's overall performance and stability.
Smart Images
Figure CN121765374A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of model training technology, and in particular to a training method and related apparatus for a case element extraction model. Background Technology
[0002] Existing technologies for text information processing, such as those for processing judicial documents, suffer from drawbacks such as low efficiency in identifying key case elements, insufficient classification accuracy, and limited applicability. Specifically, they are difficult to automate the accurate identification of key case elements for each sentence in judicial documents, relying heavily on manual extraction, which is time-consuming and error-prone, and cannot meet the demand for comprehensive and accurate element extraction in such scenarios. Summary of the Invention
[0003] To address the aforementioned issues, this application provides a training method and related apparatus for a case element extraction model, which can improve the accuracy and comprehensiveness of case element extraction.
[0004] According to one aspect of the embodiments of this application, a training method for a case element extraction model is proposed, the method comprising: The labeled dataset is determined, which includes a first dataset belonging to a first type of label, a second dataset belonging to a second type of label, and a third dataset belonging to a third type of label; The pre-trained initial model is iteratively trained based on the first dataset, the second dataset, and the third dataset to obtain the trained case element extraction model. The training steps for each iteration of the pre-trained initial model are as follows: The text information to be recognized is input into the pre-trained initial model to obtain the prediction result output by the pre-trained initial model; The evaluation metrics for the current iteration round are determined based on the prediction results and the labeled dataset. If the evaluation metric of the current iteration is lower than the preset benchmark evaluation metric, then the pre-trained initial model will be trained for the next iteration.
[0005] In the above scheme, determining the evaluation metric for the current iteration based on the prediction result and the labeled dataset includes: Based on the prediction results and the labeled dataset, a first evaluation value is determined to measure the coverage of the labels predicted by the pre-trained initial model in the multi-label classification task. Based on the prediction results and the labeled dataset, a second evaluation value is determined to measure the accuracy of the labels predicted by the pre-trained initial model in the multi-label classification task. A third evaluation value is determined based on the first evaluation value and the second evaluation value to measure the performance of the pre-trained initial model in the multi-label classification task; The evaluation index for the current iteration round is determined based on the first evaluation value, the second evaluation value, and the third evaluation value.
[0006] In the above scheme, the first evaluation value is the global recall rate. The step of determining the first evaluation value, based on the prediction results and the labeled dataset, to measure the coverage of labels predicted by the pre-trained initial model in the multi-label classification task includes: Determine the first prediction result for each label in the prediction results, and determine the recall rate for each label based on each first prediction result and the labeled dataset; The average recall rate of each of the aforementioned tags is taken as the global recall rate.
[0007] In the above scheme, the second evaluation value is the global accuracy. The step of determining the second evaluation value, based on the prediction results and the labeled dataset, to measure the accuracy of the labels predicted by the pre-trained initial model in the multi-label classification task includes: Determine a second prediction result for each label in the prediction results, and determine the precision of each label based on each second prediction result and the labeled dataset; The average of the precision rates of each of the aforementioned labels is taken as the global precision rate.
[0008] In the above scheme, determining a third evaluation value based on the first evaluation value and the second evaluation value to measure the performance of the pre-trained initial model in the multi-label classification task includes: The third evaluation value is obtained by taking a weighted average of the first evaluation value and the second evaluation value.
[0009] In the above scheme, the preset benchmark evaluation metrics include preset benchmark recall, preset benchmark precision, and preset benchmark evaluation value. If the evaluation metric of the current iteration is lower than the preset benchmark evaluation metric, then the pre-trained initial model is trained for the next iteration, including: If the first evaluation value is lower than the preset baseline recall rate, then the pre-trained initial model will be trained for the next iteration. If the second evaluation value is lower than the preset benchmark accuracy, then the pre-trained initial model will be trained for the next iteration. If the third evaluation value is lower than the preset benchmark evaluation value, then the pre-trained initial model will be trained for the next iteration.
[0010] According to one aspect of the embodiments of this application, a training device for a case element extraction model is proposed, the device comprising: A determining unit is used to determine the labeled dataset, which includes a first dataset belonging to a first type of label, a second dataset belonging to a second type of label, and a third dataset belonging to a third type of label; The training unit is used to iteratively train the pre-trained initial model based on the first dataset, the second dataset, and the third dataset to obtain the trained case element extraction model. The training steps for each iteration of the pre-trained initial model are as follows: The text information to be recognized is input into the pre-trained initial model to obtain the prediction result output by the pre-trained initial model; The evaluation metrics for the current iteration round are determined based on the prediction results and the labeled dataset. If the evaluation metric of the current iteration is lower than the preset benchmark evaluation metric, then the pre-trained initial model will be trained for the next iteration.
[0011] According to one aspect of the embodiments of this application, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the training method for the case element extraction model as described above.
[0012] According to one aspect of the embodiments of this application, a computer program product is provided, the computer program product including a computer program, the computer program being read and executed by a processor of an electronic device, causing the electronic device to perform the training method for the case element extraction model as described above.
[0013] The beneficial effects of this application are as follows: This application uses labeled datasets to iteratively train a pre-trained initial model. In each iteration, the training steps determine the evaluation index based on the predicted results and labeled datasets. If the evaluation index is lower than the preset benchmark evaluation index, the pre-trained initial model continues to be trained for the next iteration. Finally, a case element extraction model that meets the preset benchmark evaluation index is output. This improves the accuracy and comprehensiveness of case element extraction. Attached Figure Description
[0014] Figure 1 This is a system architecture diagram of the training method for the case element extraction model provided in the embodiments of this application. Figure 2 A flowchart illustrating the training method for the case element extraction model provided in this application embodiment; Figure 3 A block diagram of the training device for the case element extraction model provided in the embodiments of this application; Figure 4 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Detailed Implementation
[0015] To enable those skilled in the art to better understand the solutions of this application, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0016] It should be noted that while some processes described in the specification, claims, and accompanying drawings include multiple steps appearing in a specific order, it should be clearly understood that these steps may not be performed in the order they appear herein, or may be performed in parallel. The step numbers are merely used to distinguish different steps and do not themselves represent any execution order. Furthermore, descriptions such as "first," "second," or "objective" in this document are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. "Multiple" in this document refers to at least two.
[0017] It is worth noting that in the specific embodiments of this application, related data such as labeled datasets and text information to be identified are involved. When the above embodiments of this application are applied to specific products or technologies, permission or consent from the target object is required, and the collection, use, and processing of related data must comply with relevant laws, regulations, and standards. For example, when an embodiment of this application needs to obtain related data such as labeled datasets and text information to be identified, separate permission or consent from the target object can be obtained through pop-up windows or redirection to a confirmation page. After obtaining separate permission or consent from the target object, the necessary labeled datasets, text information to be identified, and other related data for enabling the embodiments of this application to operate normally can then be obtained.
[0018] Please see Figure 1 , Figure 1 This is a system architecture diagram of the multi-vehicle platooning method provided in this application embodiment. It includes a terminal 140, an Internet connection 130, a gateway 120, a server 110, etc.
[0019] Terminal 140 can take various forms, including desktop computers, laptops, PDAs (personal digital assistants), mobile phones, vehicle terminals, and dedicated terminals. Furthermore, it can be a single device or a collection of multiple devices. For example, multiple desktop computers can be interconnected via a local area network, sharing a single monitor to work collaboratively, forming a single terminal 140. Terminal 140 can communicate with the Internet 130 via wired or wireless means to exchange data.
[0020] Server 110 refers to a computer system capable of providing certain services to terminal 140. Compared to ordinary terminal 140, server 110 has higher requirements in terms of stability, security, and performance. Server 110 can be a single high-performance computer in a network platform, a cluster of multiple high-performance computers, a portion of a single high-performance computer (e.g., a virtual machine), or a combination of portions of multiple high-performance computers (e.g., virtual machines). Server 110 can also communicate with the Internet 130 via wired or wireless means to exchange data.
[0021] Gateway 120, also known as an internetwork connector or protocol converter, is a computer system or device that acts as a translator, enabling network interconnection at the transport layer. It bridges the gap between two systems using different communication protocols, data formats, languages, or even completely different architectures. Gateways can also provide filtering and security functions. Messages sent from terminal 140 to server 110 are forwarded to the corresponding server 110 via gateway 120. Messages sent from server 110 to terminal 140 are also forwarded to the corresponding terminal 140 via gateway 120. The following provides a detailed description of the specific implementation methods of the embodiments of this application: Please see Figure 2 , Figure 2 This is a flowchart illustrating the training method for the case element extraction model provided in the embodiments of this application. Figure 2 The training methods for the case element extraction model shown include: Step 210: Determine the labeled dataset, which includes a first dataset belonging to the first type of label, a second dataset belonging to the second type of label, and a third dataset belonging to the third type of label; Step 220: Iteratively train the pre-trained initial model based on the first dataset, the second dataset, and the third dataset to obtain the trained case element extraction model; The training steps for each iteration of the pre-trained initial model are as follows: The text information to be recognized is input into the pre-trained initial model to obtain the prediction result output by the pre-trained initial model; The evaluation metrics for the current iteration round are determined based on the prediction results and the labeled dataset. If the evaluation metric of the current iteration is lower than the preset benchmark evaluation metric, then the pre-trained initial model will be trained for the next iteration.
[0022] The following is a detailed explanation of steps 210-220: In step 210, a labeled dataset is determined. The labeled dataset includes a first dataset belonging to a first type of label, a second dataset belonging to a second type of label, and a third dataset belonging to a third type of label. Here, the first type of label can specifically be a category label for marriage and family. Under the first type of label, there are multiple first sub-labels, such as having children after marriage, raising children with limited capacity for civil conduct, having joint property of the couple, paying child support, dividing real estate, separation after marriage, paying child support monthly, granting divorce, having joint debts of the couple, premarital personal property, failing to fulfill family obligations, having children out of wedlock, providing appropriate assistance, failing to fulfill a divorce agreement, damages, separation due to incompatibility for more than two years, children living with the non-custodial parent, and personal property after marriage, etc.
[0023] The second type of label can be specifically a category label for labor disputes. Under the second type of label, there are multiple second sub-labels, such as termination of labor relations, payment of wages, payment of economic compensation, failure to pay full wages, existence of labor relations, no labor contract signed, signing of labor contract, payment of overtime wages, payment of double wages for failure to sign a labor contract, payment of work injury compensation, failure to file a labor arbitration, failure to pay compensation for illegal termination of labor relations, economic layoffs, failure to pay bonuses, illegal collection of money from workers, special types of work, payment of death benefits, funeral allowances, and survivor's pensions, early notice of termination by the employer, loss of legal person status, and existence of a mediation agreement, etc.
[0024] The third type of label can be specifically categorized as a loan relationship, such as joint guarantee, interest payment, entering into a guarantee contract, having a written repayment commitment, invalid / revoked / terminated guarantee contract, refusal to perform repayment, exemption of the guarantor from guarantee liability, the guarantor not assuming guarantee liability, the pledgor being a company, failure to provide the loan on the agreed date / amount, multiple loans, debtor transferring debt, unclear agreed interest rate, and so on.
[0025] Here, it is necessary to analyze each sentence in the text of legal documents, identify key case elements, and classify them according to a case element system designed by domain experts. Three datasets from different domains were used: a marriage and family dataset (dataset 1), a labor dispute dataset (dataset 2), and a loan contract dataset (dataset 3). Each training dataset consists of a case description fragment from a legal document, where each sentence is labeled with a corresponding category label (the number of category labels for each sentence is variable), thus completing the labeled dataset.
[0026] In step 220, the pre-trained initial model is iteratively trained based on the first dataset, the second dataset, and the third dataset to obtain the trained case element extraction model.
[0027] Specifically, the training steps for each iteration of the pre-trained initial model are as follows: The text information to be recognized is input into the pre-trained initial model to obtain the prediction result output by the pre-trained initial model; The evaluation metrics for the current iteration round are determined based on the prediction results and the labeled dataset. If the evaluation metric of the current iteration is lower than the preset benchmark evaluation metric, then the pre-trained initial model will be trained for the next iteration.
[0028] In some embodiments, determining the evaluation metric for the current iteration based on the prediction result and the labeled dataset includes: Based on the prediction results and the labeled dataset, a first evaluation value is determined to measure the coverage of the labels predicted by the pre-trained initial model in the multi-label classification task. Based on the prediction results and the labeled dataset, a second evaluation value is determined to measure the accuracy of the labels predicted by the pre-trained initial model in the multi-label classification task. A third evaluation value is determined based on the first evaluation value and the second evaluation value to measure the performance of the pre-trained initial model in the multi-label classification task; The evaluation index for the current iteration round is determined based on the first evaluation value, the second evaluation value, and the third evaluation value.
[0029] Here, the first evaluation value is the global recall rate. The step of determining the first evaluation value, based on the prediction results and the labeled dataset, to measure the coverage of labels predicted by the pre-trained initial model in the multi-label classification task includes: Determine the first prediction result for each label in the prediction results, and determine the recall rate for each label based on each first prediction result and the labeled dataset; The average recall rate of each of the aforementioned tags is taken as the global recall rate.
[0030] Specifically, the pre-trained model is trained using a labeled dataset, employing a binary classification cross-entropy loss function, and iteratively optimized using the backpropagation algorithm. During training, hyperparameter tuning is required, including learning rate, batch size, and number of iterations.
[0031] Recall, or recall rate, measures the coverage of labels predicted by a model in a multi-label classification task. Recall is the proportion of correctly predicted positive samples out of all true positive samples. In multi-label classification tasks, the recall rate needs to be calculated separately for each label, and the average of the recall rates for all labels is used as the final recall rate. A higher recall rate indicates that the model can better cover the actual labels, and the better the model's performance. The calculation formula is: .
[0032] In this context, TP is the number of true cases where the model predicts the label and the label actually exists; FP is the number of false negatives where the model fails to predict the label.
[0033] A real-world example: For a specific case element label, the model correctly predicts the existence of that element, and the element does indeed exist in the labeled data. For instance, the model identifies the element "pay child support" in a sentence of a legal document, and human annotation also confirms that the sentence does indeed contain this element.
[0034] False negatives: For a specific label, the model fails to predict the element (i.e., predicts "not present"), but the element actually exists in the labeled data. For example, a legal document may contain the phrase "granted divorce" in its label, but the model fails to recognize it (missed judgment).
[0035] False positives: For a specific label, the model incorrectly predicts the existence of the element, but the element does not actually exist in the labeled data. For example, the model judges a sentence to contain "there is a labor relationship", but the sentence does not actually involve a labor relationship (false positive).
[0036] In multi-label classification tasks, recall needs to be calculated for each label separately, and then the average of the recall rates of all labels is taken as the final recall rate. When using this method, a recall rate of ≥0.90 indicates that the model has excellent label coverage and extremely low false negative risk, making it suitable for scenarios sensitive to "false negative cost". Here, 0.9 is used as the preset baseline recall rate. For example, in the extraction of obligation-related elements (such as child support payment and loan repayment) in legal documents, to avoid missing key responsibility-related labels, a recall rate between 0.80 and 0.90 is considered good, and the false negative rate is acceptable. When it is below 0.70, the model is prone to missing important labels, and the coverage ability needs to be improved through data augmentation or model fine-tuning. For example, in the report, the chinese_lert_base_L-12_H-768_A-12_tf model achieved a recall rate of 0.97 for the label "married with children" in DV1, demonstrating an extremely low false negative rate for this type of core element.
[0037] In some embodiments, the second evaluation value is the global precision, and determining the second evaluation value based on the prediction results and the labeled dataset to measure the accuracy of the labels predicted by the pre-trained initial model in a multi-label classification task includes: Determine a second prediction result for each label in the prediction results, and determine the precision of each label based on each second prediction result and the labeled dataset; The average of the precision rates of each of the aforementioned labels is taken as the global precision rate.
[0038] Here, Precision refers to the accuracy of the labels predicted by the model in a multi-label classification task. Precision is the proportion of correctly predicted positive samples out of all predicted positive samples. In multi-label classification tasks, the precision needs to be calculated for each label separately, and the average of the precision for all labels is taken as the final precision. A higher precision indicates more accurate label predictions and better model performance. The calculation formula is: In multi-label classification tasks, the precision needs to be calculated for each label separately, and then the average value is taken as the final precision. During evaluation, a precision ≥ 0.95 means that the model's accuracy in predicting labels is excellent, with a very low risk of misjudgment, making it suitable for scenarios sensitive to "misjudgment costs". Here, 0.95 is used as the preset benchmark precision, such as the extraction of key elements in legal documents (e.g., loan amount, custody ownership), to avoid misjudging non-existent elements as existing ones. A precision between 0.90 and 0.95 is considered good, with a controllable misjudgment rate. When it is below 0.85, the model's false positive rate is too high, requiring optimization of feature extraction or adjustment of the classification threshold. For example, in the report, the two models had a precision exceeding 0.95 for most core labels (e.g., DV9 "granting divorce" and LB3 "paying economic compensation"), indicating that the accuracy of key element prediction is reliable.
[0039] Furthermore, the third evaluation metric can be represented by the F1 score, which ranges from 0 to 1. A higher value indicates better model performance, demonstrating more accurate identification of multiple labels. It is used to measure the model's performance in multi-label classification tasks. F1 is the weighted harmonic mean of precision and recall, comprehensively considering the balance between the two. The calculation formula is: In evaluation, models with an F1 score ≥ 0.90 exhibit excellent overall performance, demonstrating a good balance between precision and recall. This makes them suitable for most multi-label classification scenarios, especially when there is a trade-off between precision and recall (e.g., some rare labels are difficult to cover while maintaining accuracy). An F1 score between 0.80 and 0.90 is considered good, requiring targeted optimization of weaker metrics. A score between 0.60 and 0.80 is considered average, necessitating a focus on improving precision or recall. A score below 0.60 indicates poor overall model performance, requiring a readjustment of the training strategy.
[0040] In summary, compared with the prior art, this application has the following beneficial effects: Improved accuracy of multi-domain element extraction, breakthrough in low-frequency and complex label recognition capabilities, optimized overall model performance and stability, and enhanced multi-domain adaptability.
[0041] Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of a training device for a case element extraction model provided in an embodiment of this application. The training device for the case element extraction model is applied to a computer device, and may include: The determining unit 301 is used to determine the labeled dataset, which includes a first dataset belonging to a first type of label, a second dataset belonging to a second type of label, and a third dataset belonging to a third type of label; Training unit 302 is used to iteratively train the pre-trained initial model based on the first dataset, the second dataset, and the third dataset to obtain the trained case element extraction model. The training steps for each iteration of the pre-trained initial model are as follows: The text information to be recognized is input into the pre-trained initial model to obtain the prediction result output by the pre-trained initial model; The evaluation metrics for the current iteration round are determined based on the prediction results and the labeled dataset. If the evaluation metric of the current iteration is lower than the preset benchmark evaluation metric, then the pre-trained initial model will be trained for the next iteration.
[0042] Reference Figure 4 , Figure 4 To implement the structural block diagram of a portion of the terminal 140 in this application embodiment, the terminal 140 includes: a radio frequency (RF) circuit 710, a memory 715, an input unit 730, a display unit 740, a sensor 750, an audio circuit 760, a wireless fidelity (WiFi) module 770, a processor 780, and a power supply 790, among other components. Those skilled in the art will understand that... Figure 4 The terminal 140 structure shown does not constitute a limitation on a mobile phone or computer, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.
[0043] The RF circuit 710 can be used to receive and transmit signals during information transmission or calls. In particular, it receives downlink information from the base station and processes it with the processor 780; in addition, it transmits uplink data to the base station.
[0044] The memory 715 can be used to store software programs and modules. The processor 780 executes various functional applications of the terminal and the training processing of the case element extraction model by running the software programs and modules stored in the memory 715.
[0045] The input unit 730 can be used to receive input numeric or character information, and to generate key signal inputs related to the terminal's settings and function control. Specifically, the input unit 730 may include a touch panel 731 and other input devices 732.
[0046] The display unit 740 can be used to display input or provided information, as well as various menus of the terminal. The display unit 740 may include a display panel 741.
[0047] Audio circuitry 760, speaker 761, and microphone 762 provide an audio interface.
[0048] In this embodiment, the processor 780 included in the terminal 140 can execute the training method of the case element extraction model of the previous embodiment.
[0049] The terminal 140 in this application embodiment includes, but is not limited to, mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, and aircraft. This application embodiment can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving.
[0050] Figure 5 This is a partial structural block diagram of a server 110 implementing an embodiment of this application. The server 110 can vary significantly due to different configurations or performance characteristics, and may include one or more central processing units (CPUs) 822 (e.g., one or more processors) and memory 832, and one or more storage media 830 (e.g., one or more mass storage devices) for storing application programs 842 or data 844. The memory 832 and storage media 830 can be temporary or persistent storage. The program stored in the storage media 830 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the server 110. Furthermore, the CPU 822 may be configured to communicate with the storage media 830 and execute the series of instruction operations in the storage media 830 on the server 110.
[0051] Server 110 may also include one or more power supplies 826, one or more wired or wireless network interfaces 850, one or more input / output interfaces 858, and / or one or more operating systems 841, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.
[0052] The central processing unit 822 in server 110 can be used to execute the training method of the case element extraction model of the present application embodiment. This application also provides a computer-readable storage medium for storing program code, which is used to execute the training method of the case element extraction model of the foregoing embodiments.
[0053] This application also provides a computer program product, which includes a computer program. The processor of a computer device reads and executes the computer program, causing the computer device to perform the training method for implementing the above-described case element extraction model.
[0054] Furthermore, the terms “comprising” and “including”, and any variations thereof, are intended to cover non-exclusive inclusion, such that a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or apparatus.
[0055] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.
[0056] It should be understood that in the description of the embodiments of this application, "multiple" means two or more, "greater than", "less than", "exceeding" etc. are understood to exclude the number itself, and "above", "below", "within" etc. are understood to include the number itself.
[0057] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.
[0058] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of this application, depending on actual needs.
[0059] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0060] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0061] It should also be understood that the various implementation methods provided in this application can be combined arbitrarily to achieve different technical effects.
[0062] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.
[0063] The above is a detailed description of the embodiments of this application. However, this application is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.
Claims
1. A method for training a case element extraction model, characterized in that, The method comprises: determining a labeled data set comprising a first data set belonging to a first type of label, a second data set belonging to a second type of label, and a third data set belonging to a third type of label; iteratively training a pre-trained initial model according to the first data set, the second data set, and the third data set to obtain a trained case element extraction model; wherein the training step of each iteration round of the pre-trained initial model is: inputting the text information to be recognized into the pre-trained initial model to obtain a prediction result output by the pre-trained initial model; determining an evaluation index of the current iteration round according to the prediction result and the labeled data set; if the evaluation index of the current iteration round is lower than a preset baseline evaluation index, then performing training of the next iteration round on the pre-trained initial model.
2. The method of claim 1, wherein, The determination of the evaluation index of the current iteration round according to the prediction result and the labeled data set comprises: determining a first evaluation value for measuring the coverage rate of the labels predicted by the pre-trained initial model in the multi-label classification task according to the prediction result and the labeled data set; determining a second evaluation value for measuring the accuracy of the labels predicted by the pre-trained initial model in the multi-label classification task according to the prediction result and the labeled data set; determining a third evaluation value for measuring the performance of the pre-trained initial model in the multi-label classification task according to the first evaluation value and the second evaluation value; determining the evaluation index of the current iteration round according to the first evaluation value, the second evaluation value, and the third evaluation value.
3. The method of claim 2, wherein, The first evaluation value is a global recall rate, and the determination of the first evaluation value for measuring the coverage rate of the labels predicted by the pre-trained initial model in the multi-label classification task according to the prediction result and the labeled data set comprises: determining a first prediction result of each label in the prediction result, and determining a recall rate of each label according to each first prediction result and the labeled data set; taking the average of the recall rates of each label as the global recall rate.
4. The method of claim 3, wherein, The second evaluation value is a global precision rate, and the determination of the second evaluation value for measuring the accuracy of the labels predicted by the pre-trained initial model in the multi-label classification task according to the prediction result and the labeled data set comprises: determining a second prediction result of each label in the prediction result, and determining a precision rate of each label according to each second prediction result and the labeled data set; taking the average of the precision rates of each label as the global precision rate. 5.The method of claim 4, wherein, The determination of the third evaluation value for measuring the performance of the pre-trained initial model in the multi-label classification task according to the first evaluation value and the second evaluation value comprises: performing weighted averaging on the first evaluation value and the second evaluation value to obtain the third evaluation value. 6.The method of claim 5, wherein, The preset reference evaluation indicators include a preset reference recall rate, a preset reference precision rate, and a preset reference evaluation value. If the evaluation indicators of the current iteration round are lower than the preset reference evaluation indicators, the pre-trained initial model is trained in the next iteration round, including: If the first evaluation value is lower than the preset reference recall rate, the pre-trained initial model is trained in the next iteration round. If the second evaluation value is lower than the preset reference precision rate, the pre-trained initial model is trained in the next iteration round. If the third evaluation value is lower than the preset reference evaluation value, the pre-trained initial model is trained in the next iteration round.
7. A device for training a case element extraction model, characterized by, The device includes: A determination unit configured to determine a labeled data set, the labeled data set including a first data set belonging to a first type of label, a second data set belonging to a second type of label, and a third data set belonging to a third type of label; A training unit configured to perform iterative training on a pre-trained initial model according to the first data set, the second data set, and the third data set to obtain a trained case element extraction model. In each iteration round of the pre-trained initial model, the training steps are as follows: Inputting the text information to be recognized into the pre-trained initial model to obtain a prediction result output by the pre-trained initial model; Determining evaluation indicators of a current iteration round according to the prediction result and the labeled data set; If the evaluation indicators of the current iteration round are lower than preset reference evaluation indicators, the pre-trained initial model is trained in the next iteration round. 8.An electronic device comprising a memory and a processor, the memory storing a computer program, wherein, The processor executes the computer program to implement the training method of the case element extraction model according to any one of claims 1 to 6.
9. A computer program product comprising a computer program, characterized in that, The computer program is read and executed by the processor of the electronic device, so that the electronic device performs the training method of the case element extraction model according to any one of claims 1 to 6. The computer program is read and executed by the processor of the electronic device, so that the electronic device performs the training method of the case element extraction model according to any one of claims 1 to 6.