Computer-aided dense point labeling method and system based on large language model
Through a computer-assisted dense point annotation method based on a large language model, confidential sentence extraction and semantic feature matching are performed on long text data, which solves the problem of poor processing of long text context relationships in the existing technology and improves the accuracy of dense point annotation. sex and efficiency.
Patent Information
- Application Number
- CN202411446310.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-16
- Publication Date
- 2025-01-21
Smart Images
Figure CN119336891A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of dense point annotation, and in particular relates to a computer-aided dense point annotation method and system based on a large language model. Background Art
[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.
[0003] The task of determining secrets is the cornerstone of ensuring national security and development, and is the starting point and core link of confidentiality work. However, in actual operation, confidentiality work faces many challenges, such as the lack of historical experience as a reference, making it difficult to accurately identify confidential points; or over-reliance on the personal knowledge of confidentiality personnel, resulting in inaccurate determination of confidential points. Therefore, exploring how to use technical means to effectively assist confidentiality personnel in overcoming these difficulties is a key issue that the current confidentiality industry urgently needs to solve.
[0004] The existing technical solution uses deep learning methods to assist in confidentiality determination, collects corpus from the confidentiality industry to train the SimCSE model, which introduces a contrastive learning method to alleviate the representation degradation problem caused by anisotropy of sentence vector representation of pre-trained models such as BERT, greatly promotes the representation effect of sentence embedding, and enables sentence vectors to contain more semantic information. Therefore, the sentence embedding representation vector obtained by this model can be searched and matched more accurately in the secret point library, thereby achieving a more accurate secret point labeling effect.
[0005] However, the inventors found that although the SimCSE model has a powerful sentence representation ability, it cannot handle long texts longer than paragraphs, and cannot handle the contextual relationship of long texts. Therefore, when the method is actually performing dense point marking, if the long text to be marked is a long text, it is necessary to segment the text into sentences or paragraphs. This leads to the following problems: There is no certain standard for text segmentation at present, and the segmentation can only be judged by experience, punctuation or word length, which makes text segmentation a very difficult task. If the segmentation is too short, the dense point information may be segmented, or a sentence may lose the semantic connection of the context; if the segmentation is too long, the dense point information in the sentence may be covered by other content, thereby causing the problem of reduced accuracy of determination; after the long text is segmented, there are often a large number of sentences and paragraphs to be marked. The prior art is to input each segmented sentence into the dense point vector library for search and matching, even if most of the text in an article does not involve dense point information. Therefore, the entire dense point annotation process may take a lot of time, because each sentence vector must calculate the cosine similarity with the vector in the dense point library. When the text is very long and the dense point vector library is large, the whole process will be very time-consuming. In addition, this method of dense point annotation lacks linguistic interpretability. Summary of the invention
[0006] The embodiment of the present invention provides a computer-assisted dense point marking method and system based on a large language model to solve the problem that the SimCSE model in the prior art has a powerful sentence representation ability but cannot handle the contextual relationship of a long text, resulting in the need to pre-segment the long text into sentences or paragraphs, causing the sentences to lose contextual semantic connections, thereby affecting the accuracy of dense point marking; and the problem that the existing solution is inefficient in directly performing dense point marking on all sentences segmented from the long text.
[0007] According to a first aspect of an embodiment of the present invention, a computer-assisted dense point labeling method based on a large language model is provided, comprising:
[0008] For the long text data to be annotated with secret points, extract the secret sentences based on the pre-trained large language model; wherein the large language model takes the long text data to be annotated as input and outputs a set of secret sentences consisting of interfering secret sentences;
[0009] For each confidential sentence in the confidential sentence set, the semantic feature vector representation of the confidential sentence is obtained using the fine-tuned natural language model;
[0010] Based on the semantic feature vector representation corresponding to each confidential sentence in the confidential sentence set, it is matched with the samples in the pre-built confidential point vector library, and based on the annotation results corresponding to the matching samples in the confidential point vector library, each confidential sentence in the confidential sentence set is annotated with confidential points to achieve confidential point annotation of long text data.
[0011] Furthermore, the language patterns, semantic information, contextual relationships and semantic connections in the long text data are captured through a pre-trained large language model, and the dense sentence extraction of the long text data is achieved based on the captured language patterns, semantic information, contextual relationships and semantic connections.
[0012] Furthermore, the training of the large language model is specifically as follows: based on historical documents in the industry, long text data in the industry and its corresponding confidential sentences are obtained to construct a training corpus set; based on the obtained training corpus set, the large language model is fine-tuned to obtain a trained large language model.
[0013] Furthermore, the matching with samples in the pre-constructed secret point vector library is specifically as follows: for each secret sentence in the secret sentence set, based on the semantic feature vector representation of the secret sentence, its similarity with each secret point marked sentence sample in the secret point vector library is calculated; and the secret point marked sentence samples that meet the similarity requirements are used as matching samples of the current secret sentence.
[0014] Furthermore, the dense point annotation is performed on each dense point sentence in the dense point sentence set based on the annotation results corresponding to the matching samples in the dense point vector library, and the specific strategy is as follows:
[0015] If there are samples with similarity greater than the preset threshold, the samples are sorted according to the similarity, and the one with the greatest similarity or several samples ranked at the top are displayed for reference by the crypter to determine the annotation information of the current crypt point sentence;
[0016] If it does not exist, the annotation information of the previous key sentence will be determined manually.
[0017] Furthermore, the construction of the dense point vector library is specifically as follows: based on the dense point sentences that have been dense point annotated, using the fine-tuned SimCSE model, the semantic feature vectors of the dense point sentences are obtained, and the dense point vector library is constructed based on the obtained semantic feature vectors of the dense point sentences.
[0018] Furthermore, the natural language model specifically adopts the SimCSE model, which is trained using a supervised method based on a pre-constructed training corpus, and the type of the large language model is Qwen2, GPT4, ChatGPT, Wenxin Yiyan or Tongyi Qianwen.
[0019] According to a second aspect of an embodiment of the present invention, a computer-aided dense point labeling system based on a large language model is provided, comprising:
[0020] A secret sentence extraction unit is used to extract secret sentences from the long text data to be annotated with secret points based on a pre-trained large language model; wherein the large language model takes the long text data to be annotated as input and outputs a set of secret sentences consisting of secret sentences;
[0021] A secret sentence feature extraction unit, which is used to obtain a semantic feature vector representation of each secret sentence in the secret sentence set by using a fine-tuned natural language model;
[0022] The dense point marking unit is used to match the semantic feature vector representation corresponding to each secret sentence in the secret sentence set with the samples in the pre-built dense point vector library, and to mark each secret sentence in the secret sentence set with dense points based on the marking results corresponding to the matching samples in the dense point vector library, so as to realize the dense point marking of long text data.
[0023] According to a third aspect of an embodiment of the present invention, there is provided an electronic device, comprising a memory, a processor, and a computer program stored and running on the memory, wherein when the processor executes the program, the computer-assisted dense point labeling method based on a large language model is implemented.
[0024] According to a fourth aspect of an embodiment of the present invention, a non-transitory computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the computer-assisted dense point labeling method based on a large language model is implemented.
[0025] Compared with the prior art, the present invention has the following beneficial effects:
[0026] (1) The present invention provides a computer-assisted secret point labeling method and system based on a large language model. The solution introduces a large language model based on the excellent sentence representation ability of the sentence embedding model SimCSE, uses industry domain corpus to fine-tune the large language model, and uses its in-depth semantic understanding ability and contextual relationship capture ability to perform preliminary secret-related sentence extraction on long texts. Through fine-tuning, the large language model has deeply grasped the semantic information of secret point sentences and can accurately capture sentences and paragraphs containing secret point semantic information in long texts, cleverly solving the problem of how to segment long texts in previous technical solutions. Through the excellent contextual understanding ability of the large language model, the secret point sentences will not be segmented and lost due to sentence segmentation;
[0027] (2) The scheme embeds the suspicious confidential sentences initially identified by the large model through the SimCSE model again, retrieves and matches the obtained sentence vectors in the dense point vector library, and obtains the final dense point labeling results. The initial extraction of the large model avoids the time loss caused by matching and searching a large number of text sentences with the dense point vector library one by one, thereby improving the efficiency of dense point labeling.
[0028] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.
[0030] Figure 1 The present invention is a flowchart of a computer-aided dense point labeling method based on a large language model described in an embodiment of the present invention. DETAILED DESCRIPTION
[0031] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.
[0032] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.
[0033] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.
[0034] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.
[0035] Embodiment 1:
[0036] like Figure 1 As shown, the purpose of this embodiment is to provide a computer-assisted dense point labeling method based on a large language model, including:
[0037] Step 1: For the long text data to be annotated with secret points, extract the secret sentences based on the pre-trained large language model; wherein the large language model takes the long text data to be annotated as input and outputs a set of secret sentences consisting of interfering secret sentences;
[0038] Specifically, step 1 includes the following processing steps:
[0039] Step (1): For the text T to be annotated, there is no need to separate the sentences. Instead, the pre-set prompt template is used to directly input the pre-trained large language model, which automatically analyzes the sentences and paragraphs in the text that may involve confidential information.
[0040] Step (2): For the input text T, the large language model returns the set of identified confidential sentences S = {s1, s2, ..., s L ,}, L is the total number of possible confidential sentences extracted by the large language model, and the sentence set is input into the next step.
[0041] In a specific implementation, the training of the large language model specifically includes the following processing steps:
[0042] Define the goal of the large language model fine-tuning task and design the format of the training data: In the scheme described in this embodiment, the goal of the large language model fine-tuning is to allow the large model to extract all possible confidential sentences and provide a basis for judgment. That is, input a text T to the large model, and the large model outputs S = {s1, s2, ..., s L}, S represents the set of suspicious key point sentences obtained through large model analysis, L represents the total number of key point sentences extracted, s i Indicates the specific sentences that may be confidential that have been extracted;
[0043] Fine-tuning strategy: Fine-tune some top-level parameters of the large language model. The specific fine-tuning method can be P-tuning v2. First, randomly initialize the parameters of this layer. And set the hyperparameters, such as learning rate (lr), number of training rounds (epochs), batch size (batch size), etc.
[0044] Data preparation: Collect the secret point text data in the industry, including but not limited to emails, reports, articles, meeting minutes, etc., and ensure that these documents contain secret point information sentences. And pre-process these data to remove noise, filter some useless text, and annotate these training corpora according to the task requirements, and define prompt templates (prompt) to guide the large model to generate correct output: articles containing secret points, annotated information is the set of secret point information sentences that should be extracted; articles that do not contain secret points are annotated as articles that are not confidential;
[0045] Model adjustment: Combine fine-tuning methods, such as P-tuning v2 fine-tuning technology, to modify the model input process to ensure that the designed prompts and their corresponding learnable vectors are added before the input of each sample;
[0046] Model fine-tuning: Use annotated datasets to fine-tune the large language model, monitor the loss function and accuracy during training, adopt early stopping strategies when appropriate, and decide when to stop training based on the performance of the validation set.
[0047] Model verification and evaluation: Evaluate model performance on the validation set, pay attention to indicators such as F1 score, precision, and recall rate, adjust hyperparameters such as prompt template and learning rate according to the verification results, and perform multiple rounds of iterations if necessary. After the evaluation of the test set, save the model parameters and obtain a large language model that has been fine-tuned.
[0048] Step 2: For each confidential sentence in the confidential sentence set, use the fine-tuned natural language model to obtain the semantic feature vector representation of the confidential sentence;
[0049] Wherein, the step 2 specifically includes the following processing process:
[0050] Step (1): For the set of secret sentences output by the large model S = {s1, s2, ..., s L ,}, input the sentence into the fine-tuned SimCSE model, for each secret sentence s i , and obtain the corresponding semantic feature vector in, Represents the value of each dimension in the semantic feature vector of the secret sentence, and M is the dimension of the semantic feature vector of the secret sentence.
[0051] Step (2): Input all sentences in the secret sentence set S into the SimCSE model to obtain the semantic feature vector set V = {v1, v2, …, v L}, L is the total number of possible secret sentences extracted by the large model.
[0052] In a specific implementation, the natural language model adopts the SimCSE model, and its fine-tuning process is trained using a pre-constructed dense point data training set.
[0053] Step 3: Based on the semantic feature vector representation corresponding to each confidential sentence in the confidential sentence set, match it with the samples in the pre-built dense point vector library, and annotate each confidential sentence in the confidential sentence set with dense points based on the annotation results corresponding to the matching samples in the dense point vector library to achieve dense point annotation of long text data.
[0054] Wherein, the step 3 specifically includes the following processing process:
[0055] Step (1): Based on the pre-built dense point vector library, the cosine similarity is directly used to calculate the similarity between the semantic feature vector of the secret sentence to be marked and the dense point sentence vector in the dense point vector library. The calculation formula is as follows:
[0056]
[0057] Among them, D i represents the i-th sentence to be marked, M j Denotes the jth dense point sentence in the dense point vector library, VD i represents the semantic vector in the sentence i to be marked, VM j Represents the semantic vector in the jth dense sentence.
[0058] Step (2): Based on the matching search results of the secret sentence to be annotated in the secret point vector library, the secret determination results are sent to the secret assigner according to the value of the similarity:
[0059] Among them, for the secret sentence to be marked, first select the top K for display based on the search results of the sentence in the secret point vector library, and then set a threshold (set to 0.8 in this embodiment), and send one or more secret points whose similarity of the search results is greater than or equal to the threshold to the given secret person; if the similarity of the top K search results is not greater than or equal to the threshold, it means that there is no marked secret point sentence in the secret point library that matches the secret point sentence to be marked, and the no matching result information will be sent to the secret person.
[0060] In the specific implementation, it is necessary to construct a dense point vector library in advance, and the construction of the dense point vector library specifically includes:
[0061] Step (1) Create and initialize the FAI SS vector library, set the vector library dimension to 512 dimensions, select the L2 norm as the preprocessing method, and normalize the input vector using the L2 norm. Add IDMap, add an identifier (ID) to each vector, and use F l at brute force search index. The vector distance measurement method uses vector inner product calculation, which is equivalent to calculating the cosine similarity between vectors.
[0062] Step (2) Use the fine-tuned SimCSE model to embed the dense point data and obtain the sentence vector containing semantic information corresponding to the dense point sentence: Input a text T to the model, and the model outputs the sentence vector v corresponding to the text = {e1, e2, …, e M ,}, M is the dimension of the sentence vector.
[0063] Step (3) takes out all the dense point sentences in the dense point library in turn, inputs them into the fine-tuned SimCSE model, and stores the sentence vector v and its corresponding identifier (ID) obtained for each dense point sentence into the FAI SS vector library for subsequent retrieval and matching of dense point annotations.
[0064] In one or more embodiments, corresponding to the method in the above embodiment, a computer-assisted dense point annotation system based on a large language model is also provided, including:
[0065] A secret sentence extraction unit is used to extract secret sentences from the long text data to be annotated with secret points based on a pre-trained large language model; wherein the large language model takes the long text data to be annotated as input and outputs a set of secret sentences consisting of secret sentences;
[0066] A secret sentence feature extraction unit, which is used to obtain a semantic feature vector representation of each secret sentence in the secret sentence set by using a fine-tuned natural language model;
[0067] The dense point marking unit is used to match the semantic feature vector representation corresponding to each secret sentence in the secret sentence set with the samples in the pre-built dense point vector library, and to mark each secret sentence in the secret sentence set with dense points based on the marking results corresponding to the matching samples in the dense point vector library, so as to realize the dense point marking of long text data.
[0068] It should be noted that the system described in this embodiment corresponds to the above method, and the relevant technical details have been described in detail in Embodiment 1, so they will not be repeated here.
[0069] In further embodiments, there is also provided:
[0070] An electronic device includes a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the method described in Embodiment 1 is performed. For the sake of brevity, no further description is given here.
[0071] It should be understood that in this embodiment, the processor may be a central processing unit CPU, and the processor may also be other general-purpose processors, digital signal processors DSP, application-specific integrated circuits ASIC, off-the-shelf programmable gate arrays FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0072] The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.
[0073] A computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the method described in embodiment 1 is completed.
[0074] The method in the first embodiment can be directly embodied as a hardware processor, or a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.
[0075] Those skilled in the art will appreciate that the units, i.e., algorithm steps, of the various examples described in the present embodiment can be implemented in electronic hardware or in a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0076] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A computer-aided dense point labeling method based on a large language model, characterized in that: include: For the long text data to be annotated with secret points, extract the secret sentences based on the pre-trained large language model; wherein the large language model takes the long text data to be annotated as input and outputs a set of secret sentences consisting of interfering secret sentences; For each confidential sentence in the confidential sentence set, the semantic feature vector representation of the confidential sentence is obtained using the fine-tuned natural language model; Based on the semantic feature vector representation corresponding to each confidential sentence in the confidential sentence set, it is matched with the samples in the pre-built confidential point vector library, and based on the annotation results corresponding to the matching samples in the confidential point vector library, each confidential sentence in the confidential sentence set is annotated with confidential points to achieve confidential point annotation of long text data.
2. A computer-aided dense point labeling method based on a large language model as claimed in claim 1, characterized in that: The language patterns, semantic information, contextual relationships and semantic connections in long text data are captured through a pre-trained large language model, and the dense sentence extraction of the long text data is achieved based on the captured language patterns, semantic information, contextual relationships and semantic connections.
3. A computer-aided dense point labeling method based on a large language model as claimed in claim 1, characterized in that: The training of the large language model is specifically as follows: based on historical documents in the industry, long text data in the industry and its corresponding confidential sentences are obtained to construct a training corpus set; The large language model is fine-tuned based on the obtained training corpus set to obtain a trained large language model.
4. The computer-aided dense point labeling method based on a large language model as claimed in claim 1, characterized in that: The matching with the samples in the pre-constructed secret point vector library is specifically as follows: for each secret sentence in the secret sentence set, based on the semantic feature vector representation of the secret sentence, the similarity between it and each secret point marked sentence sample in the secret point vector library is calculated; and the secret point marked sentence samples that meet the similarity requirements are used as matching samples of the current secret sentence.
5. The computer-aided dense point labeling method based on a large language model as claimed in claim 1, characterized in that: The dense point annotation is performed on each dense point sentence in the dense point sentence set based on the annotation results corresponding to the matching samples in the dense point vector library, and the specific strategy is as follows: If there are samples with similarity greater than the preset threshold, the samples are sorted according to the similarity, and the one with the greatest similarity or several samples ranked at the top are displayed for reference by the crypter to determine the annotation information of the current crypt point sentence; If it does not exist, the annotation information of the previous key sentence will be determined manually.
6. A computer-aided dense point labeling method based on a large language model as claimed in claim 1, characterized in that: The construction of the dense point vector library is specifically as follows: based on dense point sentences that have been dense point marked, using a fine-tuned SimCSE model, obtaining semantic feature vectors of dense point sentences, and realizing the construction of the dense point vector library based on the obtained semantic feature vectors of dense point sentences.
7. The computer-aided dense point labeling method based on a large language model as claimed in claim 1, characterized in that: The natural language model specifically adopts the SimCSE model, which is trained using a supervised method based on a pre-constructed training corpus. The type of the large language model is Qwen2, GPT4, ChatGPT, Wenxin Yiyan or Tongyi Qianwen.
8. A computer-aided dense point annotation system based on a large language model, characterized in that: include: A secret sentence extraction unit is used to extract secret sentences from the long text data to be annotated with secret points based on a pre-trained large language model; wherein the large language model takes the long text data to be annotated as input and outputs a set of secret sentences consisting of secret sentences; A secret sentence feature extraction unit, which is used to obtain a semantic feature vector representation of each secret sentence in the secret sentence set by using a fine-tuned natural language model; The dense point marking unit is used to match the semantic feature vector representation corresponding to each secret sentence in the secret sentence set with the samples in the pre-built dense point vector library, and to mark each secret sentence in the secret sentence set with dense points based on the marking results corresponding to the matching samples in the dense point vector library, so as to realize the dense point marking of long text data.
9. An electronic device comprising a memory, a processor and a computer program stored and running on the memory, characterized in that: When the processor executes the program, the computer-assisted dense point labeling method based on a large language model as described in any one of claims 1 to 7 is implemented.
10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, a computer-assisted dense point labeling method based on a large language model as described in any one of claims 1 to 7 is implemented.