Large language model training data labeling method and device, computer device

Abstract

The application relates to a large language model training data labeling method and device, computer equipment, a readable storage medium and a computer program product. The method comprises the following steps: in response to a data labeling instruction of a shipping business contract sample, a data labeling page comprising a contract content display area and an audit data labeling area is displayed; the audit data labeling area comprises an audit data adjustment interface; the contract audit result obtained by the contract audit large language model by adopting the contract audit rule matched with each contract clause to audit the content of each sample contract slice is displayed in the audit data adjustment interface; in response to a labeling operation on the contract audit result, a new contract audit result is displayed; in response to a determination operation on the contract audit result, a mapping relationship among the sample contract slice content, the contract audit rule and the contract audit result is established, and model training data is obtained. The method can improve the labeling efficiency of the model training data of the contract audit large language model.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, computer device, readable storage medium, and computer program product for annotating training data of a large language model. Background Technology

[0002] With the rapid development of artificial intelligence, AI large language models are also widely used in contract review. When training large language models for contract review, especially when fine-tuning them, question-answer pair data is often required.

[0003] In traditional technologies, when users need to fine-tune the training of a large language model for contract review, they often need to access a structured file of QA pairs data (e.g., an Excel spreadsheet containing multiple QA pairs) and manually adjust or correct the structured file of the QA pairs data in order to fine-tune the training of the large language model for contract review using this structured file.

[0004] However, structured files often record a large number of QA pairs. Manually adjusting the structured files of QA pair data can make the annotation efficiency of the training data of the contract review language model low, which is not conducive to improving the fine-tuning efficiency of the contract review language model. Summary of the Invention

[0005] Based on this, it is necessary to provide a method, apparatus, computer equipment, readable storage medium, and computer program product for labeling training data of a large language model that can improve the labeling efficiency of training data for contract review large language models, addressing the aforementioned technical problems.

[0006] Firstly, this application provides a method for labeling training data for a large language model, the method comprising the following steps:

[0007] In response to a data annotation instruction on a shipping business contract sample, a data annotation page is displayed; the data annotation page includes a contract content display area and a review data annotation area; the review data annotation area includes a review data adjustment interface corresponding to the content of each sample contract slice in the shipping business contract sample; the content of the sample contract slice corresponds to each contract clause in the shipping business contract sample;

[0008] The contract review results are displayed in the review data adjustment interface; the contract review results are obtained by the contract review big language model using contract review rules that match each of the contract terms to review the content of each of the sample contract slices; the review data adjustment interface is related to the sample contract slice content currently displayed in the contract content display area;

[0009] In response to the annotation operation on the contract review result, a new contract review result is displayed;

[0010] In response to the determination of the contract review result, a mapping relationship is established between the sample contract slice content, the contract review rules and the contract review result to obtain model training data. The model training data is used to fine-tune the contract review big language model.

[0011] In one embodiment, the contract review result includes a risk type identifier and a text describing the review status;

[0012] The risk type identifier is the result obtained by the contract review big language model classifying the risk type of the corresponding sample contract slice content based on the review status description text;

[0013] The description text of the review status is the contract review big language model, which is a description of the contract review results based on the contract review rules for the corresponding sample contract slice content.

[0014] In one embodiment, the audit data adjustment interface is configured with an editing entry for the contract audit result. The editing entry includes a text editing box for displaying the audit status description text. The step of displaying a new contract audit result in response to an annotation operation on the contract audit result includes:

[0015] The text editing box receives the editing operation of the situation description text;

[0016] In response to the editing operation of the situation description text, the edited situation description text is displayed in the text editing box as the new contract review result;

[0017] The editing entry also includes a drop-down menu for displaying the risk type identifier. The display of new contract review results in response to the annotation operation on the contract review results includes:

[0018] In response to a trigger operation on the drop-down menu, multiple candidate risk identifiers are displayed through the candidate list of the drop-down menu;

[0019] In response to a triggering operation on a target risk identifier among the plurality of candidate risk identifiers, the target risk identifier is displayed in the drop-down menu as the new contract review result.

[0020] In one embodiment, the audit data annotation area displays a modification positioning control, and the method further includes:

[0021] In response to a selection operation on the target slice content of the contract content display area, the target slice content selected by the selection operation is highlighted; the target slice content is a slice content of at least one sentence in the contract clause that matches the contract review rules;

[0022] In response to the triggering operation of the modified positioning control, an association relationship is established between the audit data adjustment interface and the target slice content. The target slice content is audited through the contract audit big language model, and the new contract audit result is displayed in the audit data adjustment interface.

[0023] In one embodiment, the contract content display area displays a dividing line, which is used to separate and display sample contract slices corresponding to different contract terms, and the target slice content is the slice content that does not include the dividing line.

[0024] In one embodiment, the audit data annotation area displays an original text location control, and the method further includes:

[0025] In response to the triggering operation of the original text positioning control, the sample contract slice content that has the association relationship with the review data annotation area is displayed in the contract content display area.

[0026] Secondly, this application provides a large language model training data annotation device, the device comprising:

[0027] The response module is used to respond to data annotation instructions for shipping business contract samples and display a data annotation page; the data annotation page includes a contract content display area and a data review annotation area; the data review annotation area includes an audit data adjustment interface corresponding to the content of each sample contract slice in the shipping business contract sample; the content of the sample contract slice corresponds to each contract clause in the shipping business contract sample;

[0028] The display module is used to display the contract review results in the review data adjustment interface; the contract review results are obtained by the contract review language model using contract review rules that match each of the contract terms to review the content of each of the sample contract slices; the review data adjustment interface is related to the sample contract slice content currently displayed in the contract content display area;

[0029] The annotation module is used to display new contract review results in response to annotation operations on the contract review results;

[0030] The determination module is used to establish a mapping relationship between the sample contract slice content, the contract review rules and the contract review result in response to the determination operation of the contract review result, and to obtain model training data. The model training data is used to fine-tune the contract review big language model.

[0031] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described large language model training data annotation method.

[0032] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0033] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0034] The aforementioned method, apparatus, computer equipment, readable storage medium, and computer program product for annotating training data for large language models, in response to data annotation instructions for shipping business contract samples, display a data annotation page including a contract content display area and a review data annotation area. This allows for a clear display of the content after slicing the original contract text and the data requiring annotation on a single page, reducing the time spent manually locating contract clauses and providing a visual interface for annotating training data for the large language model in contract review. Furthermore, the review data adjustment interface in the review data annotation area displays the results obtained by the large language model in reviewing the content of each sample contract slice using contract review rules matched to each contract clause. The same review results can provide intelligent auxiliary references for data annotation; then, in response to the annotation operation of the contract review results, the new contract review results are displayed so that the intelligently generated contract review results can be visualized and annotated; then, in response to the confirmation operation of the contract review results, a mapping relationship is established between the sample contract slice content, contract review rules and contract review results, to obtain model training data for fine-tuning the contract review language model, providing real and accurate feedback samples for the fine-tuning training of the contract review language model, thereby ensuring the accuracy of the training data and the consistency with the manual annotation confirmation, and improving the annotation efficiency of the model training data of the contract review language model. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is a diagram illustrating the application environment of a large language model training data annotation method in one embodiment.

[0037] Figure 2 This is a flowchart illustrating a method for labeling training data for a large language model in one embodiment.

[0038] Figure 3 This is a schematic diagram of a data annotation page in one embodiment;

[0039] Figure 4 This is a structural block diagram of a large language model training data annotation device in one embodiment;

[0040] Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0042] The large language model training data annotation method provided in this application embodiment can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on a cloud or other network server. Terminal 102 can respond to data annotation instructions for shipping business contract samples, displaying a data annotation page including a contract content display area and an audit data annotation area. The audit data annotation area includes an audit data adjustment interface corresponding to the audit data of each sample contract slice content corresponding to each contract clause in the shipping business contract sample. Terminal 102 can display the contract audit results obtained by the contract audit language model using contract audit rules matched to each contract clause to audit the content of each sample contract slice in the audit data adjustment interface. The audit data adjustment interface is related to the sample contract slice content currently displayed in the contract content display area. Terminal 102 can respond to annotation operations on the contract audit results, displaying new contract audit results. Terminal 102 can respond to confirmation operations on the contract audit results, establishing a mapping relationship between the sample contract slice content, contract audit rules, and contract audit results, obtaining model training data for fine-tuning the contract audit language model. The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, etc. The server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0043] In one exemplary embodiment, such as Figure 2 As shown, a method for labeling training data for a large language model is provided. This embodiment applies this method to... Figure 1 The computer device 104 in the example is used for illustration. In this embodiment, the method includes the following steps S202 to S208. Wherein:

[0044] Step S202: In response to the data annotation instruction for the shipping business contract sample, the data annotation page is displayed.

[0045] like Figure 3 As shown, the data annotation page includes a contract content display area and a review data annotation area; the review data annotation area includes an interface for adjusting the review data corresponding to the content of each sample contract slice in the shipping business contract sample. The sample contract slice content corresponds to each contract clause in the shipping business contract sample.

[0046] In practical applications, users (such as data labelers and contract reviewers) can access the contract auxiliary review system installed on terminal 102 and import shipping business contract samples that need to be labeled into the system. The contract auxiliary review system will slice the shipping business contract samples according to each contract clause and create new data labeling tasks for the content of each sample contract slice. After the user accesses the data labeling function of the contract auxiliary review system, terminal 102 can respond to the data labeling instruction for the shipping business contract samples and display a data labeling page that includes a contract content display area and a data labeling review area.

[0047] Step S204: Display the contract review results in the review data adjustment interface.

[0048] The contract review result is obtained by the contract review language model using contract review rules matched to each contract clause to review the content of each sample contract slice. For example, the contract review rules may include rule source, rule ID, review topic, rule category, review perspective, rule content, etc.

[0049] As an example, contract review results may include risk type identifiers and review status description text. The risk type identifier is the result of the contract review language model classifying the risk types of the corresponding sample contract slices based on the review status description text; for example, it may include identifiers such as risky, no risk, and missing risk.

[0050] The review status description text is a large language model for contract review, which explains the review results of the corresponding sample contract slices (i.e. contract fragments to be reviewed) based on the contract review rules. For example, it includes other relevant clauses of the contract fragment to be reviewed and the review topic, whether the contract fragment to be reviewed and the review topic are related, the location of relevant contract clauses to be reviewed, the review results for the contract fragment to be reviewed, risk warnings, etc.

[0051] The audit data adjustment interface is related to the sample contract slice currently displayed in the contract content display area. It can be understood that the contract clauses corresponding to the contract audit results displayed in the audit data adjustment interface correspond one-to-one with the sample contract slice currently displayed in the contract content display area.

[0052] In practice, after the user enters the data annotation function of the contract auxiliary review system, the contract auxiliary review system will conduct a preliminary review of the content of each sample contract slice of the shipping business contract sample through the contract review big language model. The terminal 102 can display the contract review results generated by the contract review big language model in the review data adjustment interface.

[0053] Step S206: In response to the annotation operation on the contract review result, display the new contract review result.

[0054] In practical applications, users can manually evaluate or correct the contract review results generated by the large language model of contract review on the review data adjustment interface, and annotate the contract review results. Terminal 102 can respond to the annotation operation of the contract review results and display the new contract review results.

[0055] Step S208: In response to the determination of the contract review result, establish the mapping relationship between the sample contract slice content, the contract review rules and the contract review result to obtain model training data.

[0056] Understandably, the model training data is used to fine-tune the large language model for contract review, which can represent the mapping relationship between the sample contract slice content, contract review rules, and contract review results.

[0057] In the specific implementation, after the user annotates the contract review results, if the user believes that the current contract review rule has been annotated, the user needs to confirm the corresponding contract review result under the review rule. The user can click the confirmation button corresponding to the review rule in the review data adjustment interface. The terminal 102 can then respond to the confirmation operation of the contract review result, establish the mapping relationship between the sample contract slice content, the contract review rule and the contract review result, and obtain the model training data for fine-tuning the contract review big language model, that is, obtain the QA pair data required for the contract review big language model.

[0058] After completing the annotation of the contract review results corresponding to all review rules, users can click the data annotation completion control on the data annotation page. Terminal 102 can respond to the triggering operation of the data annotation completion control to batch confirm the annotated contract review results under each review rule and obtain model training data.

[0059] The aforementioned large language model training data annotation method, in response to data annotation instructions for shipping business contract samples, displays a data annotation page including a contract content display area and a review data annotation area. This allows for a clear display of the content after slicing the original contract text and the data requiring annotation on the same page, reducing the time spent manually locating contract clauses and providing a visual interface for annotating training data for the contract review large language model. Furthermore, the review data adjustment interface in the review data annotation area displays the contract review results obtained by the contract review large language model using contract review rules matched to each contract clause to review the content of each sample contract slice. This provides a basis for data annotation. The annotation provides intelligently generated auxiliary references; then, in response to the annotation operation of the contract review results, it displays the new contract review results to visualize and annotate the intelligently generated contract review results; then, in response to the confirmation operation of the contract review results, it establishes a mapping relationship between the sample contract slice content, contract review rules, and contract review results, obtaining model training data for fine-tuning the large language model of contract review, providing real and accurate feedback samples for the fine-tuning training of the large language model of contract review, thereby ensuring the accuracy of the training data and the consistency with the manual annotation confirmation, and improving the annotation efficiency of the model training data of the large language model of contract review.

[0060] In one exemplary embodiment, the audit data adjustment interface may be configured with an editing entry for the contract audit results, and the editing entry may include a text editing box for displaying a description of the audit status.

[0061] In response to the annotation operation on the contract review result, a new contract review result is displayed, including: receiving the editing operation of the situation description text through the text editing box; and displaying the edited situation description text in the text editing box as the new contract review result in response to the editing operation of the situation description text.

[0062] In the specific implementation, the review status description text, i.e., the explanation of the contract review result, is the main output of the contract review language model and the main location for data annotation. Users can manually evaluate the contract review results generated by the model. If the sample contract slice content displayed in the status description text is correctly positioned, but the status description text contains errors, such as incorrect clause positioning, missing clauses, or incorrect review results, users can edit the erroneous status description text in the text editing box. Terminal 102 can receive the editing operation of the status description text through the text editing box; in response to the editing operation, the edited status description text is displayed in the text editing box as the new contract review result.

[0063] As an example, the editing entry also includes a drop-down menu for displaying risk type identifiers. In response to the annotation operation of the contract review result, a new contract review result is displayed, including: in response to the trigger operation of the drop-down menu, displaying multiple candidate risk identifiers through the candidate list of the drop-down menu; and in response to the trigger operation of the target risk identifier among the multiple candidate risk identifiers, displaying the target risk identifier in the drop-down menu as the new contract review result.

[0064] In its implementation, the contract review language model classifies and identifies the risk types of corresponding sample contract slices based on the generated review status description text. The contract auxiliary review system can then automatically fill in the identified risk type identifiers. However, the contract review language model's identification may contain errors. Therefore, users need to edit the review status description text based on the actual situation and correct the risk types automatically identified by the contract review language model. When the user clicks a drop-down menu displaying risk type identifiers, terminal 102 can respond to the drop-down menu trigger operation by displaying multiple candidate risk identifiers, such as risk present, no risk, and missing risk, through the candidate list in the drop-down menu. After the user selects the actual target risk identifier, terminal 102 can respond to the trigger operation of the target risk identifier among the multiple candidate risk identifiers and display the target risk identifier in the drop-down menu as the new contract review result.

[0065] The technical solution of this embodiment provides an entry point for editing erroneous contract review results generated by the contract review big language model by configuring an editing entry point for the contract review results in the review data adjustment interface. Furthermore, the edited description text is displayed in the text editing box in the editing entry point, and the target risk identifier is displayed in the drop-down menu in the editing entry point. Thus, the erroneous contract review results can be modified and edited, thereby realizing the annotation operation of the contract review results.

[0066] In an exemplary embodiment, the audit data annotation area displays a modification positioning control. The method further includes: in response to a selection operation on the target slice content of the contract content display area, highlighting the target slice content selected by the selection operation; in response to a trigger operation on the modification positioning control, establishing an association between the audit data adjustment interface and the target slice content, auditing the target slice content through the contract audit big language model, and displaying the new contract audit result in the audit data adjustment interface.

[0067] The target slice content can be at least one sentence from a contract clause that matches the contract review rules. As an example, the contract content display area can display separator lines to separate sample contract slice content corresponding to different contract clauses. The target slice content is the slice content without the separator lines. Therefore, it can be understood that the target slice content selected by the user must be at least a complete sentence from a contract clause; it can span paragraphs but cannot span clauses, meaning the target slice content cannot cross separator lines.

[0068] In practical applications, if a user's assessment reveals that the sample contract slice content displayed in the situation description text is mislocated—that is, the sample contract slice content displayed in the situation description text does not correspond to the contract clauses matching the contract review rules—or if the user's assessment reveals that the contract auxiliary review system misjudged that the contract clauses corresponding to the contract review rules are missing in the sample contract slice content, the user can find and select the correct clauses in the sample contract slice content that correspond to the contract review rules in the contract content display area. Terminal 102 can respond to the selection operation of the target slice content in the contract content display area by highlighting the selected target slice content. After the user clicks the modification positioning control displayed in the review data annotation area, terminal 102 can respond to the trigger operation of the modification positioning control, establish the association between the review data adjustment interface and the target slice content, and review the target slice content through the contract review big language model. The new contract review result is displayed in the review data adjustment interface, and the user can then evaluate and confirm the new contract review result.

[0069] The technical solution of this embodiment establishes an association between the audit data adjustment interface and the selected target slice content in response to the trigger operation of the modified positioning control. This enables the incorrectly positioned slice content to be modified into the target slice content. Then, the target slice content is audited through the contract audit big language model, and the new contract audit result is displayed in the audit data adjustment interface. This enables the re-auditing of the slice content after the positioning is modified and provides a new contract audit result that requires manual annotation.

[0070] In one exemplary embodiment, the audit data annotation area displays a text location control, and the method further includes: in response to a trigger operation on the text location control, displaying sample contract slice content that is associated with the audit data annotation area in the contract content display area.

[0071] In the specific implementation, if the user needs to locate the contract clause corresponding to the current contract review rule displayed in the review data annotation area within the original contract text, after the user clicks the original text positioning control displayed in the review data annotation area, the terminal 102 can respond to the trigger operation of the original text positioning control, and the contract auxiliary review system can automatically jump to the corresponding sample contract slice content and display the sample contract slice content that is related to the review data annotation area in the contract content display area.

[0072] It should be noted that users can also locate the original text in the contract content display area by using the "Ctrl+F" search method. The terminal 102 can respond to the Ctrl+F trigger operation, display the search input area, and respond to the search content entered in the search input area that is related to the audit data annotation area, display the search content in the contract content display area to locate the corresponding sample contract slice content.

[0073] The technical solution of this embodiment, in response to the triggering operation of the original text positioning control, displays sample contract slice content that is related to the review data annotation area in the contract content display area, thereby realizing the original text positioning of the sample contract slice content displayed in the review data annotation area, thus clearly providing data annotation personnel with the correspondence between the review results and the original contract text.

[0074] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0075] Based on the same inventive concept, this application also provides a large language model training data annotation apparatus for implementing the large language model training data annotation method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations of one or more large language model training data annotation apparatus embodiments provided below can be found in the limitations of the large language model training data annotation method described above, and will not be repeated here.

[0076] In an exemplary embodiment, as shown in FIG4, a large language model training data annotation apparatus is provided, comprising:

[0077] The response module 410 is used to respond to the data annotation instruction for the shipping business contract sample and display the data annotation page; the data annotation page includes a contract content display area and an audit data annotation area; the audit data annotation area includes an audit data adjustment interface corresponding to the content of each sample contract slice in the shipping business contract sample; the content of the sample contract slice corresponds to each contract clause in the shipping business contract sample.

[0078] Display module 420 is used to display the contract review results in the review data adjustment interface; the contract review results are obtained by the contract review language model using contract review rules that match each contract clause to review the content of each sample contract slice; the review data adjustment interface is related to the sample contract slice content currently displayed in the contract content display area.

[0079] The annotation module 430 is used to display the new contract review results in response to annotation operations on the contract review results.

[0080] The determination module 440 is used to respond to the determination operation of the contract review result, establish the mapping relationship between the sample contract slice content, the contract review rules and the contract review result, and obtain model training data. The model training data is used to fine-tune the large language model of contract review.

[0081] In one embodiment, the annotation module 430 is further configured to receive an editing operation on the situation description text via a text editing box; in response to the editing operation on the situation description text, the edited situation description text is displayed in the text editing box as a new contract review result.

[0082] In one embodiment, the labeling module 430 is further configured to, in response to a trigger operation on the drop-down menu, display multiple candidate risk labels through the candidate list of the drop-down menu; and, in response to a trigger operation on the target risk label among the multiple candidate risk labels, display the target risk label in the drop-down menu as a new contract review result.

[0083] In one embodiment, the device further includes a modification positioning module 450, specifically configured to, in response to a selection operation of the target slice content in the contract content display area, highlight the target slice content selected by the selection operation; the target slice content is a slice content of at least one sentence in the contract clauses that match the contract review rules; in response to a trigger operation of the modification positioning control, establish an association between the review data adjustment interface and the target slice content, review the target slice content through the contract review big language model, and display the new contract review result in the review data adjustment interface.

[0084] In one embodiment, the device further includes a text positioning module 460, specifically used to display sample contract slice content that is related to the review data annotation area in the contract content display area in response to a trigger operation of the text positioning control.

[0085] Each module in the aforementioned large language model training data annotation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0086] In an exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram is shown in Figure 5. The computer device includes a processor, memory, input / output interface (I / O), communication interface, display unit, and input device. The processor, memory, and I / O interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the I / O interface. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The I / O interface of the computer device is used for exchanging information between the processor and external devices. The communication interface of the computer device is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a method for labeling training data for a large language model. The display unit of this computer device is used to form a visually visible image, and can be a display screen, a projection device, or a virtual reality imaging device.

[0087] Those skilled in the art will understand that the structure shown in Figure 5 is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or may combine certain components, or may have different component arrangements.

[0088] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the various embodiments of the above-described method for labeling training data for a large language model.

[0089] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the various embodiments of the above-described method for labeling training data for a large language model.

[0090] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps described in the embodiments of the method for annotating training data for a large language model.

[0091] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0092] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic resistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence processors, etc., and are not limited to these.

[0093] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0094] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for labeling training data for a large language model, characterized in that, The method includes: In response to a data annotation instruction on a shipping business contract sample, a data annotation page is displayed; the data annotation page includes a contract content display area and a review data annotation area; the review data annotation area includes a review data adjustment interface corresponding to the content of each sample contract slice in the shipping business contract sample; the content of the sample contract slice corresponds to each contract clause in the shipping business contract sample; The contract review results are displayed in the review data adjustment interface; the contract review results are obtained by the contract review big language model using contract review rules that match each of the contract terms to review the content of each of the sample contract slices; the review data adjustment interface is related to the sample contract slice content currently displayed in the contract content display area; In response to the annotation operation on the contract review result, a new contract review result is displayed; In response to the determination of the contract review result, a mapping relationship is established between the sample contract slice content, the contract review rules and the contract review result to obtain model training data. The model training data is used to fine-tune the contract review big language model.

2. The method according to claim 1, characterized in that, The contract review results include risk type identification and a description of the review status. The risk type identifier is the result obtained by the contract review big language model classifying the risk type of the corresponding sample contract slice content based on the review status description text; The description text of the review status is the contract review big language model, which is a description of the contract review results based on the contract review rules for the corresponding sample contract slice content.

3. The method according to claim 2, characterized in that, The audit data adjustment interface is configured with an editing entry for the contract audit results. This editing entry includes a text editing box for displaying a description of the audit status. The process of displaying a new contract audit result in response to annotating the contract audit results includes: The text editing box receives the editing operation of the situation description text; In response to the editing operation of the situation description text, the edited situation description text is displayed in the text editing box as the new contract review result; The editing entry also includes a drop-down menu for displaying the risk type identifier. The display of new contract review results in response to the annotation operation on the contract review results includes: In response to a trigger operation on the drop-down menu, multiple candidate risk identifiers are displayed through the candidate list of the drop-down menu; In response to a triggering operation on a target risk identifier among the plurality of candidate risk identifiers, the target risk identifier is displayed in the drop-down menu as the new contract review result.

4. The method according to claim 2, characterized in that, The audit data annotation area displays a modification positioning control, and the method further includes: In response to a selection operation on the target slice content of the contract content display area, the target slice content selected by the selection operation is highlighted; the target slice content is a slice content of at least one sentence in the contract clause that matches the contract review rules; In response to the triggering operation of the modified positioning control, an association relationship is established between the audit data adjustment interface and the target slice content. The target slice content is audited through the contract audit big language model, and the new contract audit result is displayed in the audit data adjustment interface.

5. The method according to claim 4, characterized in that, The contract content display area is marked with dividing lines, which are used to separate and display sample contract slices corresponding to different contract terms. The target slice content is the slice content that does not include the dividing lines.

6. The method according to claim 1, characterized in that, The audit data annotation area displays a text location control, and the method further includes: In response to the triggering operation of the original text positioning control, the sample contract slice content that has the association relationship with the review data annotation area is displayed in the contract content display area.

7. A large language model training data annotation device, characterized in that, The device includes: The response module is used to respond to data annotation instructions for shipping business contract samples and display a data annotation page; the data annotation page includes a contract content display area and a data review annotation area; the data review annotation area includes an audit data adjustment interface corresponding to the content of each sample contract slice in the shipping business contract sample; the content of the sample contract slice corresponds to each contract clause in the shipping business contract sample; The display module is used to display the contract review results in the review data adjustment interface; the contract review results are obtained by the contract review language model using contract review rules that match each of the contract terms to review the content of each of the sample contract slices; the review data adjustment interface is related to the sample contract slice content currently displayed in the contract content display area; The annotation module is used to display new contract review results in response to annotation operations on the contract review results; The determination module is used to establish a mapping relationship between the sample contract slice content, the contract review rules and the contract review result in response to the determination operation of the contract review result, and to obtain model training data. The model training data is used to fine-tune the contract review big language model.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images