Document review method and device, computer readable storage medium and electronic equipment

Abstract

The invention provides a document review method and device, a computer readable storage medium and electronic equipment. Comprising the steps of obtaining a training document, performing semantic information extraction and classification on review opinions of the training document, determining a pseudo tag corresponding to each review opinion, splitting the training document to obtain a target training document, and performing semantic feature extraction on the split training document to obtain a training document feature set; training the initial weak supervision network model by adopting the training document feature set and the target training document to obtain a trained weak supervision network model; the method comprises the steps of obtaining a to-be-examined document, splitting the to-be-examined document, performing semantic feature extraction to obtain a to-be-examined document feature set, and examining each clause of the to-be-examined document feature set by adopting a weak supervision network model to generate an examination document. The problem that in the prior art, secondary labeling is carried out after sentence splitting is carried out on a document, and consequently the manpower and material resource cost is high is solved.

Description

Technical Field

[0001] The present application relates to the technical field of document review, and in particular, to a document review method, device, computer-readable storage medium, and electronic device. Background Art

[0002] Generally speaking, legal review refers to the activity of conducting compliance risk review on important elements such as contracts and case information. Legal review generally requires the participation of professional legal personnel and is a time-consuming and laborious process. During the legal review of documents such as contracts, legal personnel will generally issue a legal review opinion after the review. However, the review opinion is usually a summary statement and will not be marked for a certain sentence or paragraph in the document. Therefore, it is often possible to obtain a large number of correspondences between documents and review opinions, but it is difficult to obtain a correspondence between a certain sentence in a document and the review opinion.

[0003] The existing method for reviewing documents is to split the documents into sentences and then perform secondary annotation, but this will consume a lot of manpower and material costs. Summary of the invention

[0004] The main purpose of the present application is to provide a document review method, device, computer-readable storage medium and electronic device, so as to at least solve the problem that the existing method of reviewing documents is to perform secondary annotation after breaking down the documents into sentences, which leads to high manpower and material costs.

[0005] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a document review method is provided, comprising: obtaining a training document with review opinions, using the LongFormer pre-training model to extract semantic information from the review opinions of the training document, obtaining semantic features corresponding to each of the review opinions, classifying the review opinions according to the semantic features, determining pseudo labels corresponding to each of the review opinions, splitting the training document according to the category of the pseudo labels to obtain multiple target training documents, wherein the target training document corresponds to one of the pseudo labels; splitting the training document into sentences, and using the LongFormer pre-training model to extract semantic information from the review opinions of the training document, obtaining semantic features corresponding to each of the review opinions, classifying the review opinions according to the semantic features, determining pseudo labels corresponding to each of the review opinions, and splitting the training document according to the category of the pseudo labels to obtain multiple target training documents, wherein the target training document corresponds to one of the pseudo labels; splitting the training document into sentences, and using the LongFormer pre-training model to extract semantic information from the review opinions of the training document, obtaining semantic features corresponding to each of the review opinions, and classifying the review opinions according to the semantic features, determining pseudo labels corresponding to each of the review opinions, and splitting the training document into sentences. Extract semantic features to obtain a training document feature set, wherein the training document feature set includes multiple sentences and semantic features corresponding to the sentences; construct an initial weakly supervised network model, and use the training document feature set and the target training document to train the initial weakly supervised network model to obtain a trained weakly supervised network model; obtain a document to be reviewed, split the document to be reviewed into sentences, extract semantic features from each sentence of the document to be reviewed, obtain a feature set of the document to be reviewed, and use the trained weakly supervised network model to review each sentence of the feature set of the document to be reviewed to generate a review document, wherein each sentence in the review document has a corresponding review opinion.

[0006] Optionally, the training document is split into sentences, and the LongFormer pre-trained model is used to extract semantic features from each sentence of the training document, including: splitting the training document into sentences using delimiters of the training document, wherein the delimiters include at least one of a paragraph separator, a line break, a period, and an exclamation mark; determining whether the number of sentences is less than or equal to a preset number threshold; and when it is determined that the number of sentences is less than or equal to the preset number threshold, using the LongFormer pre-trained model to extract semantic features from each sentence of the training document.

[0007] Optionally, a document to be reviewed is obtained, the document to be reviewed is split into sentences, semantic features are extracted for each sentence of the document to be reviewed, a feature set of the document to be reviewed is obtained, and the trained weakly supervised network model is used to review each sentence of the feature set of the document to be reviewed to generate a review document, wherein each sentence in the review document has a corresponding review opinion, including: obtaining the document to be reviewed, using the separator of the document to be reviewed to process the document to be reviewed into sentences, and determining whether the number of sentences in the document to be reviewed is greater than a preset number threshold; after determining the number of sentences in the document to be reviewed, When the number of the sentences is greater than the preset number threshold, the document to be reviewed is input into the weakly supervised network model using a sliding window method to review each of the sentences in the document to be reviewed, and the pseudo-label corresponding to each sentence and the predicted value of the pseudo-label are obtained, wherein the sliding window of the sliding window method is equal to the preset number threshold, and the sliding step is less than the preset number threshold; the sentences whose predicted value of the pseudo-label corresponding to each sentence is greater than the preset threshold are selected to obtain multiple target sentences, and the review opinions of each target sentence are determined according to the pseudo-labels corresponding to the multiple target sentences to generate the review document.

[0008] Optionally, after determining whether the number of sentences in the document to be reviewed is greater than a preset number threshold, the method further includes: when it is determined that the number of sentences in the document to be reviewed is less than the preset number threshold, using preset sentences to fill the document to be reviewed so that the number of sentences in the document to be reviewed after the filling process is equal to the preset number threshold, wherein the preset sentences are sentences without semantic features; using the weakly supervised network model to review each sentence of the document to be reviewed after the filling process to obtain the pseudo-label corresponding to each sentence and the predicted value of the pseudo-label; selecting the sentence whose predicted value of the pseudo-label corresponding to each sentence is greater than the preset threshold to obtain a plurality of target sentences, and determining the review opinion of each target sentence according to the pseudo-labels corresponding to the plurality of target sentences to generate the review document.

[0009] Optionally, after determining whether the number of sentences in the document to be reviewed is greater than a preset number threshold, the method further includes: when it is determined that the number of sentences in the document to be reviewed is equal to the preset number threshold, using the weakly supervised network model to review each sentence of the document to be reviewed to obtain the pseudo-label corresponding to each sentence and the predicted value of the pseudo-label; selecting the sentences whose predicted value of the pseudo-label corresponding to each sentence is greater than the preset threshold to obtain a plurality of target sentences, and determining the review opinion of each target sentence according to the pseudo-labels corresponding to the plurality of target sentences to generate the review document.

[0010] Optionally, classifying the review opinions according to the semantic features includes: using a DBSCAN algorithm to classify the review opinions according to the semantic features.

[0011] Optionally, the initial weakly supervised network model is trained using the training document feature set and the target training document to obtain a trained weakly supervised network model, including: using the training document feature set as the input of the initial weakly supervised network model, performing feature conversion on the training document feature set to obtain a document feature matrix, and then inputting the document feature matrix into a sentence branch module and a category branch module respectively, the sentence branch module outputs a first pseudo-label matrix, and the category branch module outputs a second pseudo-label matrix, the first pseudo-label matrix and the second pseudo-label matrix are both matrices formed by multiple sentences with pseudo-labels, the first pseudo-label matrix and the second pseudo-label matrix are fused to obtain a fused matrix, and the training document feature set is used as the input of the initial weakly supervised network model, and the training document feature set is converted into a document feature matrix. The pseudo-labels corresponding to the target training documents and the fusion matrix are subjected to classification loss processing to optimize the initial weakly supervised network model to obtain the trained weakly supervised network model, wherein the trained weakly supervised network model includes a feature conversion module and a category branch module, the feature conversion module is used to perform feature conversion on the input feature set of the document to be reviewed to obtain the document feature matrix, the document feature matrix includes a plurality of the sentences and semantic features corresponding to the sentences, the category branch module is used to match the pseudo-label most relevant to each sentence according to the semantic features of each sentence in the document feature matrix, and output the second pseudo-label matrix as the output of the trained weakly supervised network model.

[0012] According to another aspect of the present application, a document review device is provided, comprising: an acquisition unit, which is used to acquire a training document with review opinions, use the LongFormer pre-training model to extract semantic information from the review opinions of the training document, obtain semantic features corresponding to each of the review opinions, classify the review opinions according to the semantic features, determine pseudo labels corresponding to each of the review opinions, split the training document according to the category of the pseudo labels, and obtain multiple target training documents, wherein the target training document corresponds to one of the pseudo labels; a splitting unit, which is used to split the training document into sentences, use the LongFormer pre-training model to perform semantic feature extraction on each of the sentences in the training document Extraction to obtain a training document feature set, wherein the training document feature set includes multiple sentences and semantic features corresponding to the sentences; a training unit, used to construct an initial weakly supervised network model, and use the training document feature set and the target training document to train the initial weakly supervised network model to obtain a trained weakly supervised network model; a processing unit, used to obtain a document to be reviewed, split the document to be reviewed into sentences, extract semantic features from each sentence of the document to be reviewed, obtain a feature set of the document to be reviewed, and use the trained weakly supervised network model to review each sentence of the feature set of the document to be reviewed to generate a review document, wherein each sentence in the review document has a corresponding review opinion.

[0013] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute any one of the document review methods.

[0014] According to another aspect of the present application, an electronic device is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for executing any one of the document review methods described.

[0015] Applying the technical solution of the present application, a training document with review opinions is obtained, and a LongFormer pre-training model is used to extract semantic information from the review opinions of the training document to obtain semantic features corresponding to each review opinion, and the review opinions are classified according to the semantic features, and pseudo labels corresponding to each review opinion are determined. The training document is split according to the category of the pseudo label to obtain multiple target training documents, wherein the target training document corresponds to a pseudo label; the training document is split into sentences, and the LongFormer pre-training model is used to extract semantic features from each sentence of the training document to obtain a training document feature set, wherein the training document feature set includes multiple sentences and semantic features corresponding to the sentences; an initial weakly supervised network model is constructed, and the initial weakly supervised network model is trained to obtain a trained weakly supervised network model, wherein the process of training the initial weakly supervised network model includes: the training document The document feature set is used as the input of the initial weakly supervised network model, and the feature transformation of the training document feature set is performed to obtain a document feature matrix, and then the document feature matrix is ​​respectively input into the sentence branch module and the category branch module, the sentence branch module outputs a first pseudo-label matrix, and the category branch module outputs a second pseudo-label matrix, the first pseudo-label matrix and the second pseudo-label matrix are both matrices formed by multiple sentences with pseudo-labels, the first pseudo-label matrix and the second pseudo-label matrix are fused to obtain a fusion matrix, and the pseudo-labels and fusion matrix corresponding to the target training document are used to perform classification loss processing; the document to be reviewed is obtained, the document to be reviewed is split into sentences, semantic features are extracted for each sentence of the document to be reviewed, and a feature set of the document to be reviewed is obtained, and each sentence of the feature set of the document to be reviewed is reviewed by the trained weakly supervised network model to generate a review document, wherein each sentence in the review document has a corresponding review opinion. By using clustering methods to obtain pseudo labels for the review opinions of existing data, the process of manual labeling of review opinions is eliminated. By adopting a network structure suitable for weakly supervised text training, each sentence in the document can be predicted to have its review opinion label, which solves the problem of existing technology that the document is split into sentences and then labeled twice, resulting in high manpower and material costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings constituting part of the present application are used to provide a further understanding of the present application. The exemplary embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0017] Figure 1 A hardware structure block diagram of a mobile terminal for executing a document review method provided in an embodiment of the present application is shown;

[0018] Figure 2A schematic diagram of a process of reviewing a document provided according to an embodiment of the present application is shown;

[0019] Figure 3 A module structure diagram of a review document provided according to an embodiment of the present application is shown;

[0020] Figure 4 A schematic diagram showing documents and review opinions provided according to an embodiment of the present application is shown;

[0021] Figure 5 A schematic diagram of unifying labels for review comments of a document provided according to an embodiment of the present application is shown;

[0022] Figure 6 A schematic diagram of splitting a document according to review opinion tags provided in an embodiment of the present application is shown;

[0023] Figure 7 A schematic diagram of a document preprocessing process according to an embodiment of the present application is shown;

[0024] Figure 8 A schematic diagram of the structure of a weakly supervised network model provided according to an embodiment of the present application is shown;

[0025] Fig. 9 A schematic diagram of a process for processing a document to be reviewed according to an embodiment of the present application is shown;

[0026] Fig.10 A schematic diagram of document sentence sliding window processing provided according to an embodiment of the present application is shown;

[0027] Fig.11 A structural block diagram of a document review device provided according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0028] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0029] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] For the convenience of description, some nouns or terms involved in the embodiments of the present application are explained below:

[0032] Full supervision: The training data is in the same format as the prediction data, and the training method can adopt the typical supervised network training method.

[0033] Weak supervision: The training data labels are weak, and deeper labels are required for prediction. For example, the training data is document categories, but the category of each sentence in the document needs to be predicted.

[0034] TF-IDF: Term Frequency-Inverse Document Index, a traditional way of representing text features.

[0035] Bert,LongFormer: A large pre-trained model in natural language processing that can be used as a feature extractor for text.

[0036] As introduced in the background technology, the method for reviewing documents in the prior art is to split the documents into sentences and then perform secondary annotation, which results in high manpower and material costs. In order to solve the problem that the method for reviewing documents in the prior art is to split the documents into sentences and then perform secondary annotation, which results in high manpower and material costs, the embodiments of the present application provide a document review method, device, computer-readable storage medium and electronic device.

[0037] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0038] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 FIG. 1 is a hardware structure block diagram of a mobile terminal of a document review method according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0039] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the document review method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, the above method is implemented. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0040] In this embodiment, a method for reviewing documents running on a mobile terminal, a computer terminal or a similar computing device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0041] Figure 2 is a flow chart of a method for reviewing a document according to an embodiment of the present application. Figure 2 As shown, the method comprises the following steps:

[0042] Step S201, obtaining a training document with review comments, using the LongFormer pre-training model to extract semantic information from the review comments of the training document, obtaining semantic features corresponding to each review comment, classifying the review comments according to the semantic features, determining pseudo labels corresponding to each review comment, splitting the training document according to the categories of the pseudo labels, and obtaining multiple target training documents, wherein a target training document corresponds to a pseudo label;

[0043] Specifically, there are many ways to extract the semantic features of the review opinion, such as using the traditional TF-IDF method to generate the feature representation of the sentence, or using the pre-trained model Bert in natural language processing to obtain the semantic representation. This method uses the LongFormer pre-trained model with stronger generalization ability to extract semantic information. On the one hand, this model has better semantic expression ability, and on the other hand, it can adapt to the input of longer sentences. Expressed in formula, for the input review opinion x, its semantic feature is expressed as f(x), and the feature dimension size is 768.

[0044] Step S202, splitting the training document into sentences, using the LongFormer pre-training model to extract semantic features from each sentence of the training document, and obtaining a training document feature set, wherein the training document feature set includes multiple sentences and semantic features corresponding to the sentences;

[0045] Specifically, after obtaining suitable training documents, the LongFormer model is used to extract the semantic features of each sentence to obtain a training document feature set. The semantic feature sets of these sentences will be used as training data for the weak supervision network.

[0046] Step S203, constructing an initial weakly supervised network model, using the training document feature set and the target training document to train the initial weakly supervised network model, to obtain a trained weakly supervised network model;

[0047] The process of training the initial weakly supervised network model includes: taking the training document feature set as the input of the initial weakly supervised network model, performing feature conversion on the training document feature set to obtain a document feature matrix, and then respectively inputting the document feature matrix into the sentence branch module and the category branch module, the sentence branch module outputs a first pseudo-label matrix, and the category branch module outputs a second pseudo-label matrix, the first pseudo-label matrix and the second pseudo-label matrix are both matrices formed by multiple sentences with pseudo-labels, fusing the first pseudo-label matrix and the second pseudo-label matrix to obtain a fusion matrix, and performing classification loss processing using the pseudo-labels corresponding to the target training documents and the fusion matrix;

[0048] Specifically, a weakly supervised network model is used to predict the review opinions of each sentence in the document. During the model training process, a sentence and category dual-branch fusion network structure is proposed to convert the document classification problem into a sentence and category fusion classification problem, solving the problem of asymmetric training prediction labels.

[0049] Step S204, obtain the document to be reviewed, split the document to be reviewed into sentences, extract semantic features of each sentence of the document to be reviewed, obtain a feature set of the document to be reviewed, use the trained weakly supervised network model to review each sentence of the feature set of the document to be reviewed, and generate a review document, wherein each sentence in the review document has a corresponding review opinion.

[0050] Specifically, a document to be reviewed is obtained, and the document to be reviewed is split into sentences according to the delimiters of the document. The LongFormer pre-trained model is used to extract semantic features of each sentence of the document to be reviewed to obtain a feature set of the document to be reviewed. The feature set of the document to be reviewed is input into the trained weakly supervised network model to obtain a review document in which each sentence has a corresponding review opinion.

[0051] Through this embodiment, a training document with review opinions is obtained, and the LongFormer pre-training model is used to extract semantic information from the review opinions of the training document to obtain semantic features corresponding to each review opinion, and the review opinions are classified according to the semantic features to determine the pseudo labels corresponding to each review opinion. The training document is split according to the category of the pseudo labels to obtain multiple target training documents, wherein the target training document corresponds to a pseudo label; the training document is split into sentences, and the LongFormer pre-training model is used to extract semantic features from each sentence of the training document to obtain a training document feature set, wherein the training document feature set includes multiple sentences and semantic features corresponding to the sentences; an initial weakly supervised network model is constructed, and the initial weakly supervised network model is trained to obtain a trained weakly supervised network model, wherein the process of training the initial weakly supervised network model includes: extracting the training document feature set from the training document feature set; and extracting the semantic features from the training document feature set. The collection is used as the input of the initial weakly supervised network model, and the feature transformation is performed on the feature set of the training document to obtain the document feature matrix, and then the document feature matrix is ​​input into the sentence branch module and the category branch module respectively. The sentence branch module outputs a first pseudo-label matrix, and the category branch module outputs a second pseudo-label matrix. The first pseudo-label matrix and the second pseudo-label matrix are both matrices formed by multiple sentences with pseudo-labels. The first pseudo-label matrix and the second pseudo-label matrix are fused to obtain a fusion matrix, and the pseudo-labels and fusion matrix corresponding to the target training document are used for classification loss processing; the document to be reviewed is obtained, the document to be reviewed is split into sentences, and semantic features are extracted for each sentence of the document to be reviewed to obtain a feature set of the document to be reviewed, and each sentence of the feature set of the document to be reviewed is reviewed by the trained weakly supervised network model to generate a review document, wherein each sentence in the review document has a corresponding review opinion. By using clustering methods to obtain pseudo labels for the review opinions of existing data, the process of manual labeling of review opinions is eliminated. By adopting a network structure suitable for weakly supervised text training, each sentence in the document can be predicted to have its review opinion label, which solves the problem of existing technology that the document is split into sentences and then labeled twice, resulting in high manpower and material costs.

[0052] In the specific implementation process, the training document is split into sentences, and the LongFormer pre-trained model is used to extract semantic features of each sentence of the training document, including: splitting the training document into sentences using the delimiters of the training document, wherein the delimiters include at least one of a paragraph separator, a line break, a period, and an exclamation mark; determining whether the number of sentences is less than or equal to a preset number threshold; when it is determined that the number of sentences is less than or equal to the preset number threshold, the LongFormer pre-trained model is used to extract semantic features of each sentence of the training document.

[0053] The sentence segmentation method adopted by this method is to use separators to segment sentences. For example, when encountering symbols such as paragraph breaks, line breaks, periods, exclamation marks, etc., they will be regarded as the beginning of a new sentence. If it is determined that the number of sentences is greater than the preset number threshold, the document will not be used as a training sample; if it is determined that the number of sentences is less than the preset number threshold, the element zero is used to fill the feature so that the number of sentences is equal to the preset number threshold.

[0054] Specifically, a document to be reviewed is obtained, the document to be reviewed is split into sentences, semantic features are extracted for each sentence of the document to be reviewed, a feature set of the document to be reviewed is obtained, and each sentence of the feature set of the document to be reviewed is reviewed and processed using a trained weakly supervised network model to generate a review document, wherein each sentence in the review document has a corresponding review opinion, including: obtaining the document to be reviewed, using the delimiter of the document to be reviewed to process the document to be reviewed, and determining whether the number of sentences in the document to be reviewed is greater than a preset number threshold; when it is determined that the number of sentences in the document to be reviewed is greater than the preset number threshold, the document to be reviewed is input into the weakly supervised network model using a sliding window method to review and process each sentence of the document to be reviewed, and pseudo-labels corresponding to each sentence and predicted values ​​of the pseudo-labels are obtained, wherein the sliding window of the sliding window method is equal to the preset number threshold, and the sliding step size is less than the preset number threshold; sentences whose predicted values ​​of pseudo-labels corresponding to each sentence are greater than the preset threshold are selected to obtain multiple target sentences, and the review opinions of each target sentence are determined according to the pseudo-labels corresponding to the multiple target sentences to generate a review document.

[0055] Among them, the preset number threshold of sentences can be set to 500, 600, 800, etc.

[0056] This method cannot directly process all sentence features as model input when the number of sentences after document segmentation is greater than the maximum number of sentences (preset number threshold). Therefore, it is proposed to use a sliding window to solve the problem of a large number of sentences. Assuming that the number of sentences after segmentation is m, n is the preset number threshold, where m>n, and the step size of the sliding window is s, all sentences can be divided into several small blocks in the form of a sliding window, and the length of each small block is n. These small blocks are sent to the existing model for prediction, and the results of the model category branch are taken as the prediction results. When sliding the window, in order to cover as much context information as possible, the sliding window step size s is usually set smaller than the small block length n.

[0057] More specifically, after determining whether the number of sentences in the document to be reviewed is greater than a preset number threshold, the method also includes: when it is determined that the number of sentences in the document to be reviewed is less than the preset number threshold, using preset sentences to fill the document to be reviewed so that the number of sentences in the document to be reviewed after the filling process is equal to the preset number threshold, wherein the preset sentences are sentences without semantic features; using a weakly supervised network model to review each sentence of the document to be reviewed after the filling process, and obtain a pseudo-label corresponding to each sentence and a predicted value of the pseudo-label; selecting sentences whose predicted values ​​of the pseudo-labels corresponding to each sentence are greater than the preset threshold to obtain multiple target sentences, and determining the review opinions of each target sentence according to the pseudo-labels corresponding to the multiple target sentences to generate a review document.

[0058] When the number of sentences in the document to be reviewed is less than the maximum number of sentences after the sentence segmentation, this method directly fills the sentence features with 0 features to reach the maximum number of sentences as the model input. The padded feature set is input into the model for prediction. In order to convert the maximum probability value of the predicted pseudo-label into an actual review opinion for prompting, a threshold is set for the probability. For example, when the predicted value pi>0.85, the i-th sentence is considered to be a review opinion that may need to be reminded, and the prompt text is the review opinion category text corresponding to pi. When pi is less than the threshold, no prompt is given.

[0059] Furthermore, after determining whether the number of sentences in the document to be reviewed is greater than a preset threshold, the method also includes: when it is determined that the number of sentences in the document to be reviewed is equal to the preset threshold, using a weakly supervised network model to review each sentence of the document to be reviewed to obtain a pseudo-label corresponding to each sentence and a predicted value of the pseudo-label; selecting sentences whose predicted values ​​of the pseudo-labels corresponding to the sentences are greater than the preset threshold to obtain multiple target sentences, and determining the review opinions of each target sentence according to the pseudo-labels corresponding to the multiple target sentences to generate a review document.

[0060] When the method determines that the number of sentences in the document to be reviewed is equal to a preset threshold, a weakly supervised network model is used to review each sentence of the document to be reviewed. In order to convert the maximum probability value of the predicted pseudo-label into an actual review opinion for prompting, a threshold is set for the probability. For example, when the predicted value pi>0.85, the i-th sentence is considered to be a review opinion that may require reminder, and the prompt text is the review opinion category text corresponding to pi. When pi is less than the threshold, no prompt is given.

[0061] Furthermore, the review opinions are classified according to semantic features, including: using a DBSCAN algorithm to classify the review opinions according to semantic features.

[0062] Since the total number of categories of review opinions cannot be determined in this method, the k-means algorithm is not considered for clustering, but the density-based DBSCAN algorithm is used. The advantage of this algorithm is that there is no need to specify the number of cluster centers, only the two most important parameters MinPts and Epsilon need to be specified. MinPts represents the minimum number of data points in a cluster, and this parameter is set to 10 here; Epsilon is used to determine the maximum distance between two points to determine whether they belong to the same class, and it is set to 6 here according to the statistical results. After clustering, C cluster centers can be obtained, each category center has at least 10 data points, and the distance between samples in the category is no more than 6.

[0063] Specifically, the initial weakly supervised network model is trained using a training document feature set and a target training document to obtain a trained weakly supervised network model, including: taking the training document feature set as the input of the initial weakly supervised network model, performing feature transformation on the training document feature set to obtain a document feature matrix, and then inputting the document feature matrix into a sentence branch module and a category branch module respectively, the sentence branch module outputs a first pseudo-label matrix, the category branch module outputs a second pseudo-label matrix, the first pseudo-label matrix and the second pseudo-label matrix are both matrices formed by multiple sentences with pseudo-labels, and the first pseudo-label matrix and the second pseudo-label matrix are fused to obtain a fused matrix. A fusion matrix is ​​prepared, and the pseudo-labels corresponding to the target training documents and the fusion matrix are used for classification loss processing to optimize the initial weakly supervised network model to obtain a trained weakly supervised network model, wherein the trained weakly supervised network model includes a feature conversion module and a category branch module, the feature conversion module is used to perform feature conversion on the input feature set of the document to be reviewed to obtain a document feature matrix, the document feature matrix includes multiple sentences and semantic features corresponding to the sentences, the category branch module is used to match the pseudo-labels most relevant to each sentence according to the semantic features of each sentence in the document feature matrix, and output a second pseudo-label matrix as the output of the trained weakly supervised network model.

[0064] In order to enable those skilled in the art to more clearly understand the technical solution of the present application, the implementation process of the document review method of the present application will be described in detail below in conjunction with specific embodiments.

[0065] Generally speaking, legal review refers to the activity of conducting compliance risk review on important elements such as contracts and case information. Legal review generally requires the participation of professional legal personnel and is a time-consuming and laborious process. The use of artificial intelligence to assist in review not only saves the labor costs of legal personnel, but also plays a complementary role in identifying risk points that are difficult to detect.

[0066] There are also many methods for reviewing documents such as contracts on the current market. Some existing technical solutions are to process the review document into sentences to form multiple single sentences, input each single sentence into the system database for matching, and obtain the sentence similarity of each single sentence; determine whether the similarity of each sentence is greater than the set threshold, and when it is greater than the set threshold, mark the single sentence as a possible risk point. Some existing technical solutions are to divide the contract to be reviewed into multiple sub-segment texts, calculate the text similarity score between the sub-segment text and the clause text in the contract sample, and use the risk warning information corresponding to the clause text with the highest text similarity score as the risk warning information of the sub-segment text. Both of the above patents convert the problem of legal review into a text matching problem, which is also a method often used in legal intelligent review. Some existing technical solutions are to construct a contract review knowledge graph by sorting out the contract review requirements, implement specific contract review rules, and thus complete the contract review task. This patent uses the knowledge graph to define multiple contents such as labels, review points, review items, review rules, risk warning information, etc., and construct a review list for one-by-one comparison.

[0067] The above-mentioned existing technical solutions can all be considered as fully supervised training processes. Full supervision means that the data type required for training is consistent with the data type during use. Labels, review points and other information are all manually defined in advance, and when in use, the review prediction of labels is also performed on these review points. This embodiment is to solve the problem of inconsistency between existing review data labels (correspondence between documents and review opinions) and prediction methods (correspondence between sentences in documents and review opinions). This application proposes a weakly supervised neural network training method to solve this problem.

[0068] This embodiment addresses the shortcomings of the prior art and has the following two improvements:

[0069] 1) In the existing review scheme, the types of training data and prediction data are consistent, and a fully supervised training scheme can be used. However, this patent proposes a weakly supervised neural network training method to solve the problem of asymmetric types of training data and prediction data.

[0070] 2) In the weakly supervised neural network training method proposed in this application, clustering is used to obtain pseudo labels for the review opinions of existing data, eliminating the process of manual annotation of the review opinions; a network structure suitable for text weak supervision training is proposed, so that each sentence in the document can be predicted to obtain its review opinion label; in the legal review prediction stage, a solution combining sliding window and model fusion is given to the problem of a large number of document sentences.

[0071] This embodiment relates to a specific document review method, such as Figure 3 As shown, it mainly includes the following three modules:

[0072] 1. Label generation module: Classify a large number of legal review opinions, generate pseudo-labels, and correspond them one by one with contract documents.

[0073] 2. Model training module: Train the weak supervision model based on the existing document data and review opinion labels, including document preprocessing, weak supervision network structure design, weak supervision model training and other steps.

[0074] 3. Legal review module: conduct a sentence-by-sentence legal review of new documents to be reviewed and provide review opinions.

[0075] The three modules specifically include the following:

[0076] 1. The label generation module includes:

[0077] In legal affairs, legal personnel often need to review the basic information of the contract and the text information (usually in the form of attachments) to form legal review opinions. These review opinions are usually a general description of the text information of a document after review; they are all legal review tips for different sentences in the document, but the specific location of these sentences is not given. In other words, the data label type we can use is the correspondence between the document and the review opinion, such as Figure 4 shown.

[0078] Because the scope of legal review is wide, the review opinions issued by legal personnel are different. Therefore, we must classify these review opinions uniformly and obtain unified labels. Here we use the clustering method for classification. The specific steps are as follows:

[0079] 1) Extraction of examination opinion features:

[0080] There are many ways to extract the semantic features of the review opinion, such as using the traditional TF-IDF method to generate the feature representation of the sentence, or using the pre-trained model Bert in natural language processing to obtain the semantic representation. In this embodiment, the LongFormer pre-trained model with stronger generalization ability is used to extract semantic information. On the one hand, the model has better semantic expression ability, and on the other hand, it can adapt to the input of longer sentences. Expressed in formula, for the input review opinion x, its semantic feature is expressed as f(x), and the dimension size is 768.

[0081] 2) Classification of examination opinion characteristics:

[0082] Since the total number of categories of review opinions cannot be determined, the k-means algorithm is not considered for clustering, but the density-based DBSCAN algorithm is used. The advantage of this algorithm is that there is no need to specify the number of cluster centers, only the two most important parameters MinPts and Epsilon need to be specified. MinPts represents the minimum number of data points in a cluster, and this parameter is set to 10 here; Epsilon is used to determine the maximum distance between two points to determine whether they belong to the same class, and it is set to 6 here according to the statistical results. After clustering, C cluster centers can be obtained, each category center has at least 10 data points, and the distance between samples in the category is no more than 6.

[0083] After completing the clustering task of the review comments, the classification of the review comments is completed. Each review comment has a globally unified pseudo-label. The total number of labels is the number of cluster centers C. Each category can be represented as [C1, C2, ..., CC]. Thus, the data labels are transformed from Figure 4 Convert to Figure 5 Unified label type.

[0084] 3) Label generation:

[0085] Since the weakly supervised network structure designed in the present invention does not support multi-category classification during the training process, it is necessary to split the document and the labels of multiple review comments again to form a one-to-one correspondence between the document and the review comment labels. The final label is as follows: Figure 6 shown.

[0086] 2. Model training module:

[0087] In getting Figure 6 After the unified data labels are obtained, the focus will be on the weakly supervised neural network model generation stage. This stage mainly includes three steps: document preprocessing, weakly supervised network structure design, and weakly supervised model training. The following will introduce them separately:

[0088] 1) The preprocessing process of the entire document is as follows Figure 7 As shown in the figure, because each sentence in the document needs to be reviewed, the document is divided into sentences. The sentence division method used here is to use separators to break sentences. For example, when encountering symbols such as paragraph breaks, line breaks, periods, exclamation marks, etc., they will be regarded as the beginning of a new sentence.

[0089] Because the total number of sentences after splitting is different in each document, a maximum number of sentences will be set (which can be set to 500 according to statistical data). Considering that the subsequent neural network requires fixed input, when the number of split sentences exceeds the maximum number, the document will not be used as a training sample.

[0090] After obtaining suitable training samples, the LongFormer model is used to extract the semantic features of each sentence. The semantic feature set of these sentences will be used as training data for the weak supervision network. Each document can be represented as The n dimension is the preset maximum number of sentences, and the f dimension represents the feature dimension. If the number of sentences after a document is split does not reach the maximum number, the feature is padded with zero elements to make its input dimension reach n*f.

[0091] 2) Weakly supervised network structure design:

[0092] This embodiment designs a weakly supervised network structure to solve the problem of inconsistent training and prediction data labels. The specific network structure is as follows: Figure 8 As shown;

[0093] Feature conversion module: The input of this network structure is the feature set x of the sentence, whose feature dimension is n*f, where n is the maximum number of sentences and f is the dimension of the sentence semantic feature. First, it passes through the feature conversion module, which contains a fully connected layer fc0 to perform feature fusion conversion on the input x, followed by an activation layer relu to convert linear features to nonlinear. The converted feature x t Its dimension is n*d, where d is 512.

[0094] Sentence and category branches: After obtaining the converted features, two branches are connected, one is the sentence branch and the other is the category branch. The sentence branch is used to select the most relevant sentence for each category, and the category branch is used to select the most relevant category for each sentence. The sentence branch consists of a fully connected layer fc_s and a normalized layer softmax_s: fc_s is used to convert the feature x t The feature dimension d is transformed to the number of categories c, that is, the output dimension after fc_s is n×c. Softmax_s normalizes the sentence dimension (that is, n in n×c) to obtain the most relevant sentence in the jth category, as shown in the following formula:

[0095]

[0096] Among them, [x s ] ij Represents the matrix x s The value of the jth category of the ith statement in , e represents the exp exponential function, and k represents the kth statement index.

[0097] The category branch also consists of a fully connected layer fc_c and a normalized layer softmax_c: fc_c also transforms the feature x tThe feature dimension of d is transformed to the number of categories c, that is, the output dimension after fc_c is also n×c. The difference is that softmax_c is normalized in the category dimension (that is, c in n×c) to obtain the most relevant category under the i-th sentence, as shown in the following formula:

[0098]

[0099] Among them, [x c ] ij Represents the matrix x c The value of the jth category of the ith statement in , e represents the exp exponential function, and k represents the kth category index.

[0100] Fusion layer: After the sentence and category branches, the normalized features x are obtained respectively. s and x c Then it is sent to the fusion layer, the purpose of which is to fuse the relevant sentences and relevant categories selected by the two branches. It includes two steps: one is to convert x s and x c Perform Hadamard product, that is, x = x s ⊙x c , to complete the fusion work; the other is to accumulate the results of the Hadamard product along the sentence dimension to facilitate the distinction of the overall category of the document, that is, This will give x e As the final feature layer for classification tasks, its dimension is 1*c.

[0101] Classification loss layer: Due to the fusion layer feature x e It can be used as a feature representation of the entire document, so the most commonly used cross entropy loss softmaxLoss can be used in the loss layer during classification. Its formula is:

[0102]

[0103] Among them, [x e ] j Represents x e The j-th category element of , C represents the total number of label categories.

[0104] 3) Weakly supervised model training:

[0105] After completing the above network structure design, the weakly labeled data obtained by document preprocessing is used, and the Adam optimizer is selected. The learning rate is set to 0.001, and the model training begins. After the model converges after 20 to 30 rounds of training, a weakly supervised model for intelligent review can be obtained.

[0106] 3. Legal review module:

[0107] The legal review module is actually the process of model prediction processing. The entire legal document review module processing process is as follows Fig. 9 shown.

[0108] Since the input of the weak supervision model mentioned in this embodiment is a feature vector set of n*f dimensions (n ​​is the maximum number of sentences, f is the feature dimension), it is also necessary to convert a document into a feature vector set of sentences during prediction, where the sentence segmentation and feature extraction methods are the same as the document preprocessing method.

[0109] When the number of sentences after the document is divided is less than the maximum number of sentences, the sentence features are directly padded with 0 features to reach the maximum number of sentences as the model input. The padded feature set is input into the model for prediction, and the result x of the category branch in the model is taken. c As the model prediction result, each row represents the probability of each sentence corresponding to the classification result. Then, by taking the maximum value along the category dimension, we can get the probability value p and its corresponding category, which can be expressed as:

[0110] p i = argmax k ([x c ] ik );

[0111] Among them, p i Indicates the maximum probability of the argmax of the classification category corresponding to the i-th clause, and k indicates the k-th category index. In order to convert the maximum probability value into an actual review opinion for prompting, a threshold is set for the probability, such as p i >0.85, the i-th sentence is considered to be a review opinion that may need to be reminded, and the prompt text is p i The corresponding review opinion category text, when p i If the value is less than the threshold, no prompt will be given.

[0112] However, when the number of sentences after the document is segmented is greater than the maximum number of sentences, it is impossible to directly process all the sentence features as the input of the model. Therefore, this embodiment proposes to use a sliding window to solve the problem of a large number of sentences. Assuming that the number of sentences after the sentence is m, where m>n, and the step size of the sliding window is s, all sentences can be divided into several small blocks in the form of a sliding window, and the length of each small block is n. These small blocks are sent to the existing model for prediction respectively, and the results of the category branch are taken as the prediction results in accordance with the method introduced in the previous paragraph. When sliding the window, in order for the input to cover as much contextual information as possible, the sliding window step size s is usually set to be smaller than the small block length n. As a result, the same sentence will be repeatedly predicted by the model, such as Fig.10 shown.

[0113] For such a repeatedly predicted sentence, this embodiment adopts the method of model result fusion as the probability prediction result p of the sentence. c The specific strategy is as follows: Assume that the sentence is repeatedly predicted by k models, generating k probability vectors p, forming a k*c matrix M, and then normalizing it using softmax after summing along the dimension of the number of models k, which can be expressed as:

[0114] In this way, the probability value of the sentence model after fusion can be obtained, and then the threshold value is set as above to determine whether the sentence needs to be prompted during review.

[0115] At this point, the corresponding legal review process has been completed for different numbers of sentences in the document.

[0116] This embodiment abstracts the asymmetry between the document labels used in training and the sentence inputs used in prediction during the legal review process into a weak supervision problem, and proposes a weak supervision network structure solution to solve the problem. In addition, in the face of a large amount of unlabeled legal review opinion data, pseudo labels are formed by clustering to save the cost of manual labeling. In the design of the weak supervision model structure, a fusion network structure with two branches of sentences and categories is proposed to convert the document classification problem into a sentence and category fusion classification problem to solve the problem of asymmetric training prediction labels. In the legal review stage, a sliding window combined with model fusion method is proposed to solve the problem of a large number of document sentences.

[0117] This embodiment forms pseudo-labels for the existing unlabeled legal review opinion data through clustering, saving the cost of manual labeling; in terms of network structure design, a fusion network structure with two branches of sentences and categories is proposed, so that each sentence in the document can be predicted to obtain its review opinion label; in the review stage, a solution of sliding window combined with model fusion is proposed to address the problem of a large number of sentences in the document, which improves the accuracy of legal review of sentences.

[0118] The embodiment of the present application also provides a document review device. It should be noted that the document review device of the embodiment of the present application can be used to execute the document review method provided by the embodiment of the present application. The device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0119] The following is an introduction to the document review device provided in the embodiment of the present application.

[0120] Fig.11Schematic diagram of a document review device according to an embodiment of the present application. Fig.11 As shown, the device includes: an acquisition unit 1101, a splitting unit 1102, a training unit 1103 and a processing unit 1104.

[0121] The acquisition unit 1101 is configured to acquire a training document with review comments, extract semantic information from the review comments of the training document using the LongFormer pre-training model, obtain semantic features corresponding to each review comment, classify the review comments according to the semantic features, determine pseudo labels corresponding to each review comment, and split the training document according to the categories of the pseudo labels to obtain multiple target training documents, wherein each target training document corresponds to a pseudo label;

[0122] A splitting unit 1102 is used to split the training document into sentences, and use the LongFormer pre-training model to extract semantic features from each sentence of the training document to obtain a training document feature set, wherein the training document feature set includes multiple sentences and semantic features corresponding to the sentences;

[0123] The training unit 1103 is used to construct an initial weakly supervised network model, and train the initial weakly supervised network model using a training document feature set and a target training document to obtain a trained weakly supervised network model;

[0124] Processing unit 1104 is used to obtain the document to be reviewed, split the document to be reviewed into sentences, extract semantic features of each sentence of the document to be reviewed, obtain a feature set of the document to be reviewed, use the trained weakly supervised network model to review each sentence of the feature set of the document to be reviewed, and generate a review document, wherein each sentence in the review document has a corresponding review opinion.

[0125] In this embodiment, pseudo labels are obtained by clustering the review opinions of the existing data, thereby eliminating the process of manual labeling of the review opinions. A network structure suitable for weakly supervised text training is adopted so that each sentence in the document can be predicted to obtain its review opinion label, thereby solving the problem of the prior art of splitting the document into sentences and then performing secondary labeling, which results in high manpower and material costs.

[0126] As an optional solution, the splitting unit includes a splitting module, a first determination module and an extraction module; the splitting module is used to split the training document into sentences using the delimiters of the training document, wherein the delimiters include at least one of a paragraph separator, a line break, a period, and an exclamation mark; the first determination module is used to determine whether the number of sentences is less than or equal to a preset number threshold; the extraction module is used to extract semantic features of each sentence of the training document using the LongFormer pre-trained model when it is determined that the number of sentences is less than or equal to the preset number threshold.

[0127] An optional solution, the processing unit includes an acquisition module, a first processing module and a second determination module; the acquisition module is used to acquire a document to be reviewed, use the delimiter of the document to be reviewed to perform sentence processing on the document to be reviewed, and determine whether the number of sentences in the document to be reviewed is greater than a preset number threshold; the first processing module is used to use a sliding window method to input the document to be reviewed into a weakly supervised network model when it is determined that the number of sentences in the document to be reviewed is greater than the preset number threshold, so as to perform review processing on each sentence of the document to be reviewed, and obtain a pseudo-label corresponding to each sentence and a predicted value of the pseudo-label, wherein the sliding window of the sliding window method is equal to the preset number threshold, and the sliding step is less than the preset number threshold; the second determination module is used to select sentences whose predicted values ​​of the pseudo-labels corresponding to each sentence are greater than the preset threshold, obtain multiple target sentences, and determine the review opinions of each target sentence according to the pseudo-labels corresponding to the multiple target sentences to generate a review document.

[0128] Among them, the preset number threshold of sentences can be set to 500, 600, 800, etc.

[0129] An optional solution, the processing unit also includes a second processing module, a third processing module and a third determination module; the second processing module is used to, after determining whether the number of sentences in the document to be reviewed is greater than a preset number threshold, use preset sentences to fill the document to be reviewed when determining that the number of sentences in the document to be reviewed is less than the preset number threshold, so that the number of sentences in the document to be reviewed after the filling process is equal to the preset number threshold, wherein the preset sentences are sentences without semantic features; the third processing module is used to use a weakly supervised network model to review each sentence of the document to be reviewed after the filling process, and obtain a pseudo-label corresponding to each sentence and a predicted value of the pseudo-label; the third determination module is used to select sentences whose predicted values ​​of the pseudo-labels corresponding to each sentence are greater than the preset threshold, obtain multiple target sentences, and determine the review opinions of each target sentence according to the pseudo-labels corresponding to the multiple target sentences to generate a review document.

[0130] An optional solution, the processing unit also includes a fourth processing module and a fourth determination module; the fourth processing module is used to, after determining whether the number of sentences in the document to be reviewed is greater than a preset number threshold, use a weakly supervised network model to review and process each sentence of the document to be reviewed when it is determined that the number of sentences in the document to be reviewed is equal to the preset number threshold, and obtain a pseudo-label corresponding to each sentence and a predicted value of the pseudo-label; the fourth determination module is used to select sentences whose predicted values ​​of the pseudo-labels corresponding to each sentence are greater than the preset threshold, obtain multiple target sentences, and determine the review opinions of each target sentence according to the pseudo-labels corresponding to the multiple target sentences to generate a review document.

[0131] In an optional solution, the acquisition unit includes a classification module for classifying the review opinions according to semantic features using a DBSCAN algorithm.

[0132] An optional solution uses a training document feature set and a target training document to train an initial weakly supervised network model to obtain a trained weakly supervised network model, including: using the training document feature set as the input of the initial weakly supervised network model, performing feature conversion on the training document feature set to obtain a document feature matrix, and then inputting the document feature matrix into a sentence branch module and a category branch module respectively, the sentence branch module outputs a first pseudo-label matrix, and the category branch module outputs a second pseudo-label matrix, the first pseudo-label matrix and the second pseudo-label matrix are both matrices formed by multiple sentences with pseudo-labels, fusing the first pseudo-label matrix and the second pseudo-label matrix to obtain a fused matrix, and performing classification loss processing using the pseudo-labels corresponding to the target training documents and the fused matrix to optimize the initial weakly supervised network model to obtain a trained weakly supervised network model.

[0133] The document review device includes a processor and a memory, and the acquisition unit, splitting unit, training unit, processing unit, etc. are all stored in the memory as program units, and the processor executes the program units stored in the memory to implement corresponding functions. The modules are all located in the same processor; or, the modules are located in different processors in any combination.

[0134] The processor includes a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be set, and the kernel parameters can be adjusted to solve the problem that the existing method of reviewing documents is to perform secondary annotation after breaking the documents into sentences, which leads to high manpower and material costs.

[0135] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0136] An embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the document review method.

[0137] An embodiment of the present invention provides a processor, which is used to run a program, wherein the program executes the document review method when running.

[0138] An embodiment of the present invention provides an electronic device, the device includes a processor, a memory, and a program stored in the memory and executable on the processor, and the processor implements a document review method when executing the program. The device in this article may be a server, a PC, a PAD, a mobile phone, etc.

[0139] The present application also provides a computer program product which, when executed on a data processing device, is suitable for executing a program for initiating at least the steps of the method for reviewing the document.

[0140] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0141] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0142] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0143] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0144] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0145] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0146] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0147] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0148] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0149] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A document review method, characterized in that: include: Obtaining a training document with review comments, extracting semantic information from the review comments of the training document using a LongFormer pre-training model, obtaining semantic features corresponding to each review comment, classifying the review comments according to the semantic features, determining pseudo labels corresponding to each review comment, splitting the training document according to the categories of the pseudo labels, and obtaining a plurality of target training documents, wherein the target training document corresponds to one of the pseudo labels; Splitting the training document into sentences, and using the LongFormer pre-training model to extract semantic features from each sentence of the training document to obtain a training document feature set, wherein the training document feature set includes multiple sentences and semantic features corresponding to the sentences; Constructing an initial weakly supervised network model, and training the initial weakly supervised network model using the training document feature set and the target training document to obtain a trained weakly supervised network model; Obtain a document to be reviewed, split the document to be reviewed into sentences, perform semantic feature extraction on each sentence of the document to be reviewed, obtain a feature set of the document to be reviewed, use the trained weakly supervised network model to review each sentence of the feature set of the document to be reviewed, and generate a review document, wherein each sentence in the review document has a corresponding review opinion.

2. The method according to claim 1, characterized in that Splitting the training document into sentences, and using the LongFormer pre-trained model to extract semantic features from each sentence of the training document, including: Splitting the training document into the sentences using a separator of the training document, wherein the separator includes at least one of a paragraph separator, a line break, a period, and an exclamation mark; Determining whether the number of the sentences is less than or equal to a preset number threshold; When it is determined that the number of the sentences is less than or equal to the preset number threshold, the LongFormer pre-training model is used to extract semantic features from each of the sentences in the training document.

3. The method according to claim 1, characterized in that Acquire a document to be reviewed, split the document to be reviewed into sentences, extract semantic features from each sentence of the document to be reviewed, obtain a feature set of the document to be reviewed, use the trained weakly supervised network model to review each sentence of the feature set of the document to be reviewed, and generate a review document, including: Acquire the document to be reviewed, use the separator of the document to be reviewed to process the document to be reviewed, and determine whether the number of the sentences in the document to be reviewed is greater than a preset number threshold; When it is determined that the number of the sentences of the document to be reviewed is greater than the preset number threshold, the document to be reviewed is input into the weakly supervised network model using a sliding window method to review each sentence of the document to be reviewed, and obtain the pseudo-label corresponding to each sentence and the predicted value of the pseudo-label, wherein the sliding window of the sliding window method is equal to the preset number threshold, and the sliding step is less than the preset number threshold; The sentences whose predicted values ​​of the pseudo-labels corresponding to the sentences are greater than a preset threshold are selected to obtain multiple target sentences, and the review opinions of each target sentence are determined according to the pseudo-labels corresponding to the multiple target sentences to generate the review document.

4. The method according to claim 3, characterized in that After determining whether the number of sentences in the document to be reviewed is greater than a preset number threshold, the method further includes: When it is determined that the number of the sentences in the document to be reviewed is less than the preset number threshold, the document to be reviewed is filled with preset sentences so that the number of the sentences in the document to be reviewed after the filling process is equal to the preset number threshold, wherein the preset sentences are sentences without semantic features; Using the weakly supervised network model to examine each sentence of the document to be examined after the filling process, to obtain the pseudo label corresponding to each sentence and the predicted value of the pseudo label; The sentences whose predicted values ​​of the pseudo-labels corresponding to the sentences are greater than the preset threshold are selected to obtain a plurality of target sentences, and the review opinions of the target sentences are determined according to the pseudo-labels corresponding to the plurality of target sentences to generate the review document.

5. The method according to claim 3, characterized in that: After determining whether the number of sentences in the document to be reviewed is greater than a preset number threshold, the method further includes: When it is determined that the number of sentences in the document to be reviewed is equal to the preset number threshold, the weakly supervised network model is used to review each sentence of the document to be reviewed to obtain the pseudo-label corresponding to each sentence and the predicted value of the pseudo-label; The sentences whose predicted values ​​of the pseudo-labels corresponding to the sentences are greater than the preset threshold are selected to obtain a plurality of target sentences, and the review opinions of the target sentences are determined according to the pseudo-labels corresponding to the plurality of target sentences to generate the review document.

6. The method according to claim 1, characterized in that The examination opinions are classified according to the semantic features, including: The DBSCAN algorithm is used to classify the review opinions according to the semantic features.

7. The method according to claim 1, characterized in that The initial weakly supervised network model is trained using the training document feature set and the target training document to obtain a trained weakly supervised network model, including: The training document feature set is used as the input of the initial weakly supervised network model, and the training document feature set is subjected to feature conversion to obtain a document feature matrix, and then the document feature matrix is ​​respectively input into a sentence branch module and a category branch module, the sentence branch module outputs a first pseudo-label matrix, and the category branch module outputs a second pseudo-label matrix, the first pseudo-label matrix and the second pseudo-label matrix are both matrices formed by multiple sentences with pseudo-labels, the first pseudo-label matrix and the second pseudo-label matrix are fused to obtain a fused matrix, and the pseudo-label corresponding to the target training document and the fused matrix are used to perform classification loss processing to optimize the initial weakly supervised network model, so as to obtain the trained weakly supervised network model, Among them, the trained weakly supervised network model includes a feature conversion module and a category branch module, the feature conversion module is used to perform feature conversion on the input feature set of the document to be reviewed to obtain the document feature matrix, the document feature matrix includes multiple sentences and semantic features corresponding to the sentences, and the category branch module is used to match the pseudo-label most relevant to each sentence according to the semantic features of each sentence in the document feature matrix, and output the second pseudo-label matrix as the output of the trained weakly supervised network model.

8. A document review device, characterized in that: include: The acquisition unit is configured to acquire a training document with review comments, extract semantic information from the review comments of the training document using a LongFormer pre-training model, obtain semantic features corresponding to each review comment, classify the review comments according to the semantic features, determine pseudo labels corresponding to each review comment, and split the training document according to the categories of the pseudo labels to obtain a plurality of target training documents, wherein each target training document corresponds to one of the pseudo labels; A splitting unit, used to split the training document into sentences, and use the LongFormer pre-training model to extract semantic features from each sentence of the training document to obtain a training document feature set, wherein the training document feature set includes multiple sentences and semantic features corresponding to the sentences; A training unit, used to construct an initial weakly supervised network model, and train the initial weakly supervised network model using the training document feature set and the target training document to obtain a trained weakly supervised network model; A processing unit is used to obtain a document to be reviewed, split the document to be reviewed into sentences, extract semantic features of each sentence of the document to be reviewed, obtain a feature set of the document to be reviewed, use the trained weakly supervised network model to review each sentence of the feature set of the document to be reviewed, and generate a review document, wherein each sentence in the review document has a corresponding review opinion.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the document review method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for performing a document review method as described in any one of claims 1 to 7.

Images