Text review method and device, electronic equipment and computer program product

Abstract

The invention discloses a text review method and device, electronic equipment and a computer program product. The method comprises the steps of obtaining a to-be-reviewed text; the to-be-examined text is examined through the currently-used text examination scheme, an initial text examination result is obtained, the currently-used text examination scheme is determined after multiple text examination schemes are evaluated, and the text examination schemes comprise a retrieval scheme and a classification scheme; and optimizing the initial text review result according to a review optimization strategy corresponding to the currently used text review scheme to obtain a final text review result. According to the method, a new text review framework is constructed by evaluating the effects of multiple different text review schemes, the multiple different text review schemes cooperate with one another, complementarity among the multiple schemes is mined, the text review effect is improved, result optimization strategies of the different text review schemes are designed, and the text review efficiency is improved. And the accuracy of text review is further improved.

Description

Technical Field

[0001] The present application relates to the field of text review technology, and in particular to a text review method, device, electronic equipment, and computer program product. Background Art

[0002] Generally speaking, text review is the process of conducting compliance and legal review of multiple elements in the text, and is a key link in ensuring business compliance and legal risk prevention and control. For example, during the legal review of contract documents by legal personnel, they will generally issue legal opinions after the review. These issued legal review opinions can provide reference for subsequent contract reviews and serve as a basis for auxiliary review.

[0003] Traditionally, this process is highly dependent on legal personnel with profound legal knowledge and rich practical experience, which is not only time-consuming and labor-intensive, but also difficult to fully capture all potential legal risk points. With the rapid development of information technology, especially the rise of artificial intelligence technology, the use of automated tools to assist or replace some manual review work has become an industry trend.

[0004] At present, a relatively reliable solution is to collect a large amount of annotated data, train a classification model to classify the text to be reviewed, and then give review opinions. However, in the production process, a large amount of annotated data is often difficult to obtain, so some low-cost solutions need to be given in the early stage of production. Most of these low-cost solutions are non-model training solutions, such as using retrieval solutions to solve text review problems, but these solutions are generally less accurate than model training solutions.

[0005] It can be seen that most existing technologies only use a single solution, which fails to effectively integrate the respective advantages of retrieval and classification models to achieve the best balance between review efficiency and accuracy, and also fails to fully consider the shortcomings of different solutions and take effective optimization measures. Therefore, how to build a comprehensive review solution that takes into account both review efficiency and accuracy has become a technical problem that needs to be solved urgently. Summary of the invention

[0006] The embodiments of the present application provide a text review method, device, electronic device, and computer program product to improve the accuracy of text review.

[0007] The present application embodiment adopts the following technical solutions:

[0008] In a first aspect, an embodiment of the present application provides a text review method, the text review method comprising:

[0009] Get the text to be reviewed;

[0010] Using a currently used text review scheme to review the text to be reviewed to obtain an initial text review result, wherein the currently used text review scheme is determined after evaluating multiple text review schemes, and the multiple text review schemes include a retrieval scheme and a classification scheme;

[0011] The initial text review result is optimized according to the review optimization strategy corresponding to the currently used text review solution to obtain the final text review result.

[0012] Optionally, the plurality of text review schemes include a retrieval scheme based on a pre-trained model, a retrieval scheme based on a text review classification model, and a classification scheme based on a text review classification model, and the text review method further includes:

[0013] Get the training dataset for the text review classification model;

[0014] When the amount of data in the training data set is greater than a preset data amount threshold, using the training data set to train a text review classification model;

[0015] Generate a retrieval scheme based on the text review classification model and a classification scheme based on the text review classification model according to the trained training text review classification model;

[0016] Evaluate a plurality of the text review solutions and determine a text review solution that can be used online.

[0017] Optionally, the evaluating the plurality of text review schemes to determine a text review scheme that can be put into use online includes:

[0018] Construct a full-cycle validation dataset;

[0019] The full-cycle verification data set is used to evaluate multiple text review solutions to determine the text review solution that can be used online.

[0020] Optionally, constructing a full-cycle verification data set includes:

[0021] Obtain the original training data set, the original validation data set, and the newly added training data;

[0022] Using the original training data set and the newly added training data to update the original verification data set, to obtain an updated verification data set;

[0023] The full-cycle validation dataset is constructed according to the original validation dataset and the updated validation dataset.

[0024] Optionally, the step of evaluating a plurality of text review solutions using the full-cycle verification data set to determine a text review solution that can be put into use online includes:

[0025] Using the pre-trained model to perform feature extraction on the data of the full-cycle verification data set and the training data set, performing retrieval and matching based on the first feature extraction result of the full-cycle verification data set and the first feature extraction result of the training data set, to obtain a first classification result of the full-cycle verification data set;

[0026] Using the text review classification model to perform feature extraction on the data of the full-cycle verification data set and the training data set, performing retrieval matching based on the second feature extraction result of the full-cycle verification data set and the second feature extraction result of the training data set, to obtain a second classification result of the full-cycle verification data set;

[0027] Using the text review classification model to classify the data of the full-cycle verification data set, to obtain a third classification result of the full-cycle verification data set;

[0028] A text review scheme that can be used online is determined based on the first classification result, the second classification result and the third classification result of the full-cycle verification data set.

[0029] Optionally, the text review method further includes:

[0030] Determining whether to trigger an update strategy of the text review classification model, wherein the update strategy includes at least one of a periodic update strategy and a transactional update strategy;

[0031] When an update strategy of a text review classification model is triggered, the text review classification model is updated.

[0032] Optionally, the currently used text review scheme is a retrieval scheme implemented based on a pre-trained model or a text review classification model, and the initial text review result is optimized according to the review optimization strategy corresponding to the currently used text review scheme to obtain the final text review result, including:

[0033] Obtaining a plurality of search results having a similarity greater than a preset threshold value among the search results obtained by using the search solution implemented based on the pre-trained model or the text review classification model;

[0034] Counting the text review categories corresponding to multiple search results that are higher than a preset similarity threshold;

[0035] The final text review results are determined based on the statistical results.

[0036] Optionally, the currently used text review scheme is a classification scheme implemented based on a text review classification model, and the initial text review result is optimized according to the review optimization strategy corresponding to the currently used text review scheme to obtain the final text review result, including:

[0037] Determine whether the category probability corresponding to the text review result obtained by the classification scheme implemented based on the text review classification model is less than a preset probability threshold;

[0038] When the category probability is less than a preset probability threshold, obtaining a search result obtained by a search scheme implemented based on a pre-trained model and a text review classification model;

[0039] The text classification results output by the text review classification model and the retrieval results obtained by the retrieval scheme implemented based on the pre-trained model and the text review classification model are integrated to determine the final text review result.

[0040] In a second aspect, an embodiment of the present application further provides a text review device, the text review device comprising:

[0041] A first acquisition unit, used to acquire the text to be reviewed;

[0042] A review unit, configured to review the text to be reviewed using a currently used text review scheme to obtain an initial text review result, wherein the currently used text review scheme is determined after evaluating a plurality of text review schemes, wherein the plurality of text review schemes include a retrieval scheme and a classification scheme;

[0043] The optimization unit is used to optimize the initial text review result according to the review optimization strategy corresponding to the currently used text review solution to obtain the final text review result.

[0044] In a third aspect, an embodiment of the present application further provides an electronic device, including:

[0045] Processor; and

[0046] A memory arranged to store computer executable instructions, which when executed cause the processor to perform any of the text review methods described above.

[0047] In a fourth aspect, an embodiment of the present application further provides a computer program product, comprising a computer program or instructions, which, when executed by a processor, implements any of the aforementioned text review methods.

[0048] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects: the text review method of the embodiments of the present application first obtains the text to be reviewed; then the text to be reviewed is reviewed using the currently used text review scheme to obtain an initial text review result, wherein the currently used text review scheme is determined after evaluating multiple text review schemes, and the multiple text review schemes include retrieval schemes and classification schemes; finally, the initial text review result is optimized according to the review optimization strategy corresponding to the currently used text review scheme to obtain the final text review result. The text review method of the embodiments of the present application constructs a new text review framework by evaluating the effects of multiple different text review schemes. Multiple different text review schemes work together to explore the complementarity between multiple schemes, improve the effect of text review, and design result optimization strategies for different text review schemes to further improve the accuracy of text review. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0050] Figure 1 A flowchart of a text review method in an embodiment of the present application;

[0051] Figure 2 A schematic diagram of a training structure of a text review classification model in an embodiment of the present application;

[0052] Figure 3 This is a schematic diagram of a multi-scheme evaluation process in an embodiment of the present application;

[0053] Figure 4 A schematic diagram of a text review process for integrating multiple solution results in an embodiment of the present application;

[0054] Figure 5 This is a schematic diagram of the overall process of text review in an embodiment of the present application;

[0055] Figure 6 This is a schematic diagram of the structure of a text review device in an embodiment of the present application;

[0056] Figure 7 This is a schematic diagram of the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0058] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0059] To facilitate the understanding of the various embodiments of the present application, the main text review schemes adopted in the prior art are first introduced:

[0060] Currently, there are many technical solutions proposed for text review on the market, which can be mainly divided into two categories: retrieval solutions and classification solutions.

[0061] Retrieval-based solutions: This type of solution usually breaks down the document to be reviewed into smaller units (such as single sentences or sub-segments), then retrieves similar or related content in the system database, and identifies potential risk points by calculating sentence or text similarity. For example, patent document CN111046142A identifies possible risky single sentences by processing sentences and comparing similar sentences in the database; patent document CN110163478A provides risk warning information for sub-segment text by calculating the similarity between the sub-segment text and the clause text in the contract sample. This type of method does not require a large amount of annotated data, and the implementation cost is relatively low, but there are certain limitations in accuracy and semantic understanding.

[0062] Classification model-based solution: Unlike the retrieval solution, the classification model solution relies on machine learning or deep learning algorithms. By training a large amount of annotated data, the model can autonomously learn the semantic features of the text, and then automatically classify and assess the risk of the document. As shown in patent document CN114860882A, this method uses a text classification model to classify and review government policy documents, significantly improving the accuracy and efficiency of the review and effectively reducing the workload of experts. However, this method has high requirements for the amount of data, and the model construction and training process is relatively complex.

[0063] Although both of the above two types of solutions have shown certain application value in the field of document review, they each have obvious technical limitations. Although the retrieval solution is low-cost, it is incapable of dealing with complex semantic relationships and identifying potential legal risks. Although the classification solution has a high accuracy rate, it is highly dependent on data and faces challenges in model adaptability adjustment and cross-domain application.

[0064] Based on this, the present application embodiment provides a text review method, such as Figure 1 As shown, a flowchart of a text review method in an embodiment of the present application is provided, and the text review method at least includes the following steps S110 to S130:

[0065] Step S110, obtaining the text to be reviewed;

[0066] When conducting text review in the embodiment of the present application, it is necessary to first obtain the text to be reviewed, where the text to be reviewed can be a contract text, a declaration material, an advertising copy, etc. The text review method in the embodiment of the present application can be applied to multiple fields related to documents, such as contract document review, submission material analysis, etc. The specific application field is not specifically limited here.

[0067] Step S120, using the currently used text review scheme to review the text to be reviewed to obtain an initial text review result, wherein the currently used text review scheme is determined after evaluating multiple text review schemes, and the multiple text review schemes include a retrieval scheme and a classification scheme.

[0068] When conducting text review, you also need to determine the text review scheme currently in use. The text review scheme used online here is determined after a comprehensive evaluation of multiple text review schemes including retrieval schemes and classification schemes. The purpose of the evaluation is to ensure that the review scheme currently in use is the most effective scheme.

[0069] Then, the text to be reviewed is reviewed using the currently used text review scheme to obtain initial review results. For example, for a retrieval scheme, multiple retrieval results based on similarity calculations can be obtained. For a classification scheme, the probability results of the text belonging to each category can be obtained.

[0070] Step S130, optimizing the initial text review result according to the review optimization strategy corresponding to the currently used text review solution to obtain a final text review result.

[0071] Taking into account that the text review scheme may face inaccurate and unclear initial text review results during the actual application stage, the embodiment of the present application needs to further adopt the review optimization strategy corresponding to the currently used text review scheme to optimize the retrieval results or classification results obtained by the above review scheme, so as to obtain more accurate text review results.

[0072] The text review method of the embodiment of the present application constructs a new text review framework by evaluating the effects of multiple different text review schemes. The multiple different text review schemes work together to explore the complementarity between the multiple schemes, improve the effect of text review, and design result optimization strategies for different text review schemes to further improve the accuracy of text review.

[0073] In some embodiments of the present application, the multiple text review schemes include a retrieval scheme based on a pre-trained model, a retrieval scheme based on a text review classification model, and a classification scheme based on a text review classification model. The text review method also includes: obtaining a training data set for the text review classification model; when the amount of data in the training data set is greater than a preset data amount threshold, using the training data set to train the text review classification model; generating a retrieval scheme based on the text review classification model and a classification scheme based on the text review classification model according to the trained training text review classification model; and evaluating the multiple text review schemes to determine a text review scheme that can be used online.

[0074] The text review scheme of the embodiment of the present application can be divided into two categories: retrieval scheme and classification scheme. The retrieval scheme can specifically include a retrieval scheme based on a pre-trained model and a retrieval scheme based on a text review classification model. The classification scheme is a classification scheme based on a text review classification model.

[0075] The retrieval scheme based on the pre-trained model can be implemented in combination with the retrieval scheme provided in the prior art. For example, the LongFormer pre-trained model with stronger generalization ability can be used as the base model (hereinafter referred to as BaseModel) for semantic information extraction. This model is similar to the Bert semantic model, but it has better semantic expression ability on the one hand, and can adapt to the input of longer sentences on the other hand. The extracted semantic information is calculated for similarity with the information stored in the database, so that the category corresponding to the most similar information is determined as the category of the current text to be reviewed.

[0076] Both the retrieval scheme based on the text review classification model and the classification scheme based on the text review classification model need to be implemented based on the text review classification model. Before training the text review classification model, you can first determine whether the current requirements for classification model training are met based on the amount of data available online. When the amount of data is insufficient, training the classification model is likely to lead to underfitting. Therefore, you can first use the conventional retrieval scheme, that is, the retrieval scheme based on BaseModel above. When the amount of data reaches a certain amount, such as 100 texts in each category and more than 10 categories in total, the classification model can be trained.

[0077] Model training refers to the process of obtaining sufficient annotated data and fine-tuning the semantic model. Annotated data mainly refers to the annotation of the text sentences to be reviewed. For example, the text content "If Party B violates this agreement, Party B shall compensate Party A for all direct losses caused thereby in accordance with the contract, unless otherwise provided by law." is labeled as a direct loss risk category.

[0078] The training of the text review classification model can be implemented on the basis of the above BaseModel, which mainly includes fine-tuning the parameters of the BaseModel and the training process of the newly added classifier. The goal of model training is to make the text review classification model adapt to the existing review data and establish classification tasks. Finally, both the retrieval scheme and the classification scheme need to be applied to the text review stage, which is carried out for each sentence in the text to be reviewed.

[0079] like Figure 2 As shown, a schematic diagram of the training structure of a text review classification model in an embodiment of the present application is provided. The network structure of the text review classification model mainly includes two parts: a base model and a classifier. The classifier is connected after the feature output layer of the base model. Two layers of fully connected layers FullyConnection can be used in the classifier, and nonlinear layers Relu are used between the fully connected layers for feature processing. The two fully connected layers are followed by a Softmax layer for classification. The loss function can use SoftmaxLoss, and the number of categories is the number of categories of review opinions.

[0080] During training, the learning rate of the base model parameters was lowered by 0.01 times for parameter fine-tuning, and the classifier was trained with complete learning rate parameters. The Adam optimization algorithm was used for optimization, and the initial learning rate was set to 0.001. The training was stopped after 30 to 40 iterations and when the accuracy of the validation set no longer changed. The trained model was used as the adapted text review classification model AdapModel.

[0081] Of course, it should be noted that the specific design of the network structure of the classifier and the setting of the learning rate can be flexibly set by those skilled in the art according to the actual training effect and training objectives, and no specific limitation is made here.

[0082] In some embodiments of the present application, the evaluating of the plurality of text review schemes to determine a text review scheme that can be used online includes: constructing a full-cycle verification data set; and using the full-cycle verification data set to evaluate the plurality of text review schemes to determine a text review scheme that can be used online.

[0083] In order to ensure the review effect of the text review scheme currently in use online, before the text review scheme is put online, the embodiment of the present application can first evaluate the review effects of the three types of schemes, namely, the retrieval scheme based on the pre-trained model, the retrieval scheme based on the text review classification model, and the classification scheme based on the text review classification model, so as to select the optimal review scheme to go online. Of course, it should be noted that after evaluating the optimal text review scheme, it does not mean that only the optimal text review scheme can be used. Other schemes can still be used for text review, but the optimal text review scheme will be used first.

[0084] When evaluating multiple text review schemes, the embodiment of the present application adopts the method of constructing a full-cycle verification set for multi-scheme evaluation. The verification set can usually reflect the effect of the model during the model training process. As the amount of available training data continues to increase, the verification set should also change. The full-cycle verification set proposed in the embodiment of the present application is updated every time the training data set of the model training changes. Verification using the full-cycle verification set can more comprehensively and accurately evaluate the review effects of multiple schemes.

[0085] In some embodiments of the present application, constructing a full-cycle verification data set includes: obtaining an original training data set, an original verification data set, and newly added training data; updating the original verification data set using the original training data set and the newly added training data to obtain an updated verification data set; and constructing the full-cycle verification data set based on the original verification data set and the updated verification data set.

[0086] When constructing a full-cycle validation set, first fix the number of validation sets to remain unchanged, such as 1,000 samples as the validation set. Then update the validation set based on the ratio of the newly added training data to the original training data. For example, if the newly added data accounts for 20% of the original data, then randomly discard 20% of the data in the original validation set, and use 200 data from the newly added data source to supplement the new validation set, ensuring that the number of validation sets remains unchanged. Of course, you can also set a maximum discard ratio of 60% to ensure that the new model effect is also applicable to the original validation set. Finally, maintain the original validation set and the updated validation set, and use these two validation sets as the full-cycle validation sets for this model training.

[0087] It should be noted that the settings of the specific ratios and other parameters involved in the above process can be flexibly set by those skilled in the art according to actual needs and are not specifically limited here.

[0088] In some embodiments of the present application, the use of the full-cycle verification dataset to evaluate multiple text review schemes and determine the text review scheme that can be used online includes: using the pre-trained model to extract features from the data of the full-cycle verification dataset and the training dataset, and performing retrieval and matching based on the first feature extraction result of the full-cycle verification dataset and the first feature extraction result of the training dataset to obtain a first classification result of the full-cycle verification dataset; using the text review classification model to extract features from the data of the full-cycle verification dataset and the training dataset, and performing retrieval and matching based on the second feature extraction result of the full-cycle verification dataset and the second feature extraction result of the training dataset to obtain a second classification result of the full-cycle verification dataset; using the text review classification model to classify the data of the full-cycle verification dataset to obtain a third classification result of the full-cycle verification dataset; and determining the text review scheme that can be used online based on the first classification result, the second classification result, and the third classification result of the full-cycle verification dataset.

[0089] like Figure 3 As shown, a schematic diagram of a multi-scheme evaluation process in an embodiment of the present application is provided. The first scheme is a scheme for retrieval using BaseModel, which uses BaseModel to extract features of the full-cycle validation set and training set, uses each feature of the validation set as a query condition to perform retrieval in the training set, takes the top K results of the retrieval, and determines which category the validation data belongs to based on the distribution of the training set labels corresponding to these K results.

[0090] The second solution is to use the text review classification model for retrieval, which is similar to the first method, except that the new model AdapModel is used to replace BaseModel for feature extraction.

[0091] The third method is to use the text review classification model for classification. Each data in the entire validation set is classified through AdapModel to obtain the predicted label of the validation set.

[0092] After obtaining the results of the validation set based on these three solutions, the method with the best overall effect is selected to go online.

[0093] In some embodiments of the present application, the text review method also includes: determining whether to trigger an update strategy for a text review classification model, the update strategy including at least one of a periodic update strategy and a transactional update strategy; and updating the text review classification model when the update strategy for the text review classification model is triggered.

[0094] Since online data is constantly changing, the model also needs to change to adapt to online data changes. The embodiment of the present application proposes two model update trigger strategies, one is periodic update and the other is transactional update.

[0095] Periodic update means that the model will be updated within a certain time period, and the latest data will be used to adaptively fine-tune the model. Because online data accumulation requires a certain time period, a period length can be set, such as once a month or so, for updating. When the specified time period is reached, the embodiment of the present application will also determine whether the amount of new data in the period meets the standard. If the amount of new data reaches a certain proportion of the number of existing training sets, it will be updated. Otherwise, no periodic update will be performed, which ensures the real-time and effectiveness of the model.

[0096] Transactional updates refer to updates that are not within the scope of periodic updates but are made when certain specific transactions occur. For example, if the number of review categories increases, the model at this time cannot adapt to the new categories and needs to be trained. Of course, when new categories are added, new model training is not performed immediately, and it takes a period of time to accumulate new category data. During this period, the online model review method can be switched from a classification model to a retrieval model to ensure adaptability to new category data, because the classification model can only give existing category results, while the retrieval model can give similarity, judge the similarity between the current object to be reviewed and the existing results, and has better generalization capabilities.

[0097] The model's adaptability to online data is ensured through periodic and transactional updates. It can be said to be an exclusive model customized for online data.

[0098] In some embodiments of the present application, the currently used text review scheme is a retrieval scheme implemented based on a pre-trained model or a text review classification model, and the initial text review result is optimized according to the review optimization strategy corresponding to the currently used text review scheme to obtain the final text review result, including: obtaining multiple retrieval results above a preset similarity threshold from the retrieval results obtained using the retrieval scheme implemented based on a pre-trained model or a text review classification model; counting the text review categories corresponding to the multiple retrieval results above the preset similarity threshold; and determining the final text review result based on the statistical results.

[0099] After the offline model training and model evaluation steps in the above-mentioned embodiments, the model used online can be determined. Online text review refers to the process of parsing and reviewing the documents to be reviewed, and its review scheme is divided into a retrieval scheme and a classification scheme.

[0100] The retrieval scheme is similar to the conventional retrieval process, which is to divide the document to be reviewed into sentences, extract semantic features sentence by sentence, search in the specified online database based on its semantic features, select the most similar one as the matching result, and take the review category corresponding to the most similar data in this database as the review result of the text sentence to be reviewed.

[0101] However, in actual search results, the first hit may not be the correct result. Therefore, the embodiment of the present application proposes a similarity voting strategy as a search result re-optimization strategy, that is, taking the top K results (such as K=10) above the similarity threshold in the search results, counting the K corresponding result categories, and taking the result with the highest category statistics as the accurate result of this search. This similarity voting strategy uses the idea that the collective is better than the individual, and is more robust to the accuracy of the search.

[0102] In some embodiments of the present application, the currently used text review scheme is a classification scheme implemented based on a text review classification model, and the initial text review result is optimized according to the review optimization strategy corresponding to the currently used text review scheme to obtain the final text review result, including: determining whether the category probability corresponding to the text review result obtained by the classification scheme implemented based on the text review classification model is less than a preset probability threshold; when the category probability is less than the preset probability threshold, obtaining the retrieval result obtained by the retrieval scheme implemented based on the pre-trained model and the text review classification model; fusing the text classification result output by the text review classification model and the retrieval result obtained by the retrieval scheme implemented based on the pre-trained model and the text review classification model to determine the final text review result.

[0103] For the classification scheme based on the classification model, the text sentence to be reviewed is directly sent to the classification model to obtain the specific category. However, in actual use, there are also cases of inaccurate classification. After analyzing the reasons, it is found that the classification results of many erroneous samples are very uniform, that is, the classification layer of the model is not sure which category these samples should be classified into. In view of this situation, the embodiment of the present application proposes a solution of multi-model result fusion to solve the problem of uniform classification results, that is, uncertain classification.

[0104] like Figure 4As shown, a schematic diagram of a text review process of multi-scheme result fusion in an embodiment of the present application is provided. The definition of uniform classification results in the embodiment of the present application is whether the maximum probability output by the classification layer of the model is lower than a certain threshold, such as the threshold is set to 0.5. When the maximum probability of the output is higher than the threshold, it can be considered that the classification layer has a high confidence in the sample, and the classification result can be directly used as the final result. If the maximum probability is lower than the threshold, it is considered that the model classification layer cannot determine the category to which the sample belongs, so the strategy of multi-scheme result fusion in the embodiment of the present application will be used.

[0105] The BaseModel search results and AdapModel search results are obtained through the search scheme described above. The AdapModel classification result is the classification result of the currently used model. The fusion of these three results adopts the following method:

[0106] Take the TopK1 result in the BaseModel search result, the TopK2 result in the AdapModel search result, and the TopK3 result in the AdapModel classification result to form a new set {TopK1, TopK2, TopK3}. Perform statistics on this new set and take the category with the highest frequency as the final result. This method also uses the idea that the collective is better than the individual, ensuring the robustness of the results.

[0107] So far, two major types of solutions applicable to online text review have been described, which are new ideas for solving practical problems. Whether it is the problem of inaccurate first-place prediction or uniform classification model results, this application provides a robust solution, which effectively improves the accuracy of text review.

[0108] To facilitate the understanding of the above embodiments of the present application, Figure 5 As shown, a schematic diagram of the overall process of text review in an embodiment of the present application is provided. First, before training the text review classification model, it is possible to determine whether the current requirements for text review classification model training are met based on the amount of data available online. If the amount of data does not meet the requirements, a conventional retrieval scheme is used for text review. If the amount of data meets the requirements, these data are used to train the text review classification model offline, and then a full-cycle verification set is used to comprehensively evaluate the effects of multiple text review schemes, thereby selecting the optimal text review scheme to go online. Since the online data set is changing, a certain update strategy can be adopted to update the text review classification model and continuously optimize the model.

[0109] After the text review solution is launched, the text review is performed using the optimal text review solution used online. Considering that the retrieval solution and classification solution may result in inaccurate review results during application, corresponding review result optimization strategies can be adopted to optimize the results and obtain the final text review results.

[0110] In summary, the main improvements of this application are:

[0111] The main improvements and technical effects of this application compared to the prior art mainly include:

[0112] 1) Most of the existing text review solutions use retrieval solutions or classification model solutions, and the complementarity of the two solutions has not been explored. However, this application adaptively combines the two solutions and integrates them into the same new text review framework, coordinates the retrieval and classification models, explores the complementarity of the two solutions, and improves the effect of text review.

[0113] 2) During the online text review process, the retrieval scheme has the problem of inaccurate first-place prediction. This application designs a similarity voting solution. The classification model scheme has the problem of fuzzy prediction (uniform classification). This application also designs a multi-model fusion solution combined with retrieval. Both of the above solutions solve the problem of inaccurate text review results.

[0114] 3) During the offline training process, there are problems such as incomplete model evaluation, unrepresentative verification set, and lack of model adaptability. This application solves the problem of inaccurate evaluation by establishing a three-model structure, constructing a full-cycle verification set to strengthen the consistency between the verification set and online data, and using periodic update and transaction update strategies to improve the adaptability of the model.

[0115] The present application embodiment also provides a text review device 600, such as Figure 6 As shown, a schematic diagram of the structure of a text review device in an embodiment of the present application is provided, wherein the text review device 600 includes: a first acquisition unit 610, a review unit 620, and an optimization unit 630, wherein:

[0116] A first acquisition unit 610 is used to acquire a text to be reviewed;

[0117] A review unit 620 is used to review the text to be reviewed using a currently used text review scheme to obtain an initial text review result, wherein the currently used text review scheme is determined after evaluating multiple text review schemes, and the multiple text review schemes include a retrieval scheme and a classification scheme;

[0118] The optimization unit 630 is used to optimize the initial text review result according to the review optimization strategy corresponding to the currently used text review solution to obtain the final text review result.

[0119] In some embodiments of the present application, the multiple text review schemes include a retrieval scheme based on a pre-trained model, a retrieval scheme based on a text review classification model, and a classification scheme based on a text review classification model. The text review device 600 also includes: a second acquisition unit, used to acquire a training data set for the text review classification model; a training unit, used to train the text review classification model using the training data set when the amount of data in the training data set is greater than a preset data amount threshold; a generation unit, used to generate a retrieval scheme based on the text review classification model and a classification scheme based on the text review classification model according to the trained training text review classification model; an evaluation unit, used to evaluate the multiple text review schemes and determine a text review scheme that can be used online.

[0120] In some embodiments of the present application, the evaluation unit is specifically used to: construct a full-cycle verification data set; use the full-cycle verification data set to evaluate multiple text review schemes to determine a text review scheme that can be used online.

[0121] In some embodiments of the present application, the evaluation unit is specifically used to: obtain an original training data set, an original verification data set, and newly added training data; update the original verification data set using the original training data set and the newly added training data to obtain an updated verification data set; and construct the full-cycle verification data set based on the original verification data set and the updated verification data set.

[0122] In some embodiments of the present application, the evaluation unit is specifically used to: use the pre-trained model to perform feature extraction on the data of the full-cycle verification data set and the training data set, and perform retrieval and matching based on the first feature extraction result of the full-cycle verification data set and the first feature extraction result of the training data set to obtain a first classification result of the full-cycle verification data set; use the text review classification model to perform feature extraction on the data of the full-cycle verification data set and the training data set, and perform retrieval and matching based on the second feature extraction result of the full-cycle verification data set and the second feature extraction result of the training data set to obtain a second classification result of the full-cycle verification data set; use the text review classification model to classify the data of the full-cycle verification data set to obtain a third classification result of the full-cycle verification data set; determine a text review scheme that can be used online based on the first classification result, the second classification result and the third classification result of the full-cycle verification data set.

[0123] In some embodiments of the present application, the text review device 600 also includes: a determination unit, used to determine whether to trigger an update strategy for a text review classification model, the update strategy including at least one of a periodic update strategy and a transactional update strategy; an update unit, used to update the text review classification model when the update strategy for the text review classification model is triggered.

[0124] In some embodiments of the present application, the currently used text review scheme is a retrieval scheme implemented based on a pre-trained model or a text review classification model, and the optimization unit 630 is specifically used to: obtain multiple retrieval results that are higher than a preset similarity threshold from the retrieval results obtained using the retrieval scheme implemented based on a pre-trained model or a text review classification model; perform statistics on the text review categories corresponding to the multiple retrieval results that are higher than the preset similarity threshold; and determine the final text review result based on the statistical results.

[0125] In some embodiments of the present application, the currently used text review scheme is a classification scheme implemented based on a text review classification model, and the optimization unit 630 is specifically used to: determine whether the category probability corresponding to the text review result obtained by the classification scheme implemented based on the text review classification model is less than a preset probability threshold; when the category probability is less than the preset probability threshold, obtain the retrieval result obtained by the retrieval scheme implemented based on the pre-trained model and the text review classification model; fuse the text classification result output by the text review classification model and the retrieval result obtained by the retrieval scheme implemented based on the pre-trained model and the text review classification model to determine the final text review result.

[0126] It can be understood that the above-mentioned text review device can implement the various steps of the text review method provided in the aforementioned embodiment, and the relevant explanations about the text review method are applicable to the text review device and will not be repeated here.

[0127] Figure 7 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Figure 7 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. Of course, the electronic device may also include hardware required for other services.

[0128] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0129] The memory is used to store the program. Specifically, the program may include a program code, and the program code includes a computer operation instruction. The memory may include a memory and a non-volatile memory, and provides instructions and data to the processor.

[0130] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a text review device at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:

[0131] Get the text to be reviewed;

[0132] Using a currently used text review scheme to review the text to be reviewed to obtain an initial text review result, wherein the currently used text review scheme is determined after evaluating multiple text review schemes, and the multiple text review schemes include a retrieval scheme and a classification scheme;

[0133] The initial text review result is optimized according to the review optimization strategy corresponding to the currently used text review solution to obtain the final text review result.

[0134] The above application Figure 1The method performed by the text review device disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or instructions in software form. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module can be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0135] The present application also provides a computer program product, including a computer program or an instruction, which, when executed by a processor, implements any of the aforementioned text review methods, and is specifically used to perform:

[0136] Get the text to be reviewed;

[0137] Using a currently used text review scheme to review the text to be reviewed to obtain an initial text review result, wherein the currently used text review scheme is determined after evaluating multiple text review schemes, and the multiple text review schemes include a retrieval scheme and a classification scheme;

[0138] The initial text review result is optimized according to the review optimization strategy corresponding to the currently used text review solution to obtain the final text review result.

[0139] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0140] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0141] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0142] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0143] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0144] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0145] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0146] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0147] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0148] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A text review method, characterized in that: The text review method comprises: Get the text to be reviewed; Using a currently used text review scheme to review the text to be reviewed to obtain an initial text review result, wherein the currently used text review scheme is determined after evaluating multiple text review schemes, and the multiple text review schemes include a retrieval scheme and a classification scheme; The initial text review result is optimized according to the review optimization strategy corresponding to the currently used text review solution to obtain the final text review result.

2. The text review method according to claim 1, characterized in that: The plurality of text review schemes include a retrieval scheme based on a pre-trained model, a retrieval scheme based on a text review classification model, and a classification scheme based on a text review classification model. The text review method further includes: Get the training dataset for the text review classification model; When the amount of data in the training data set is greater than a preset data amount threshold, using the training data set to train a text review classification model; Generate a retrieval scheme based on the text review classification model and a classification scheme based on the text review classification model according to the trained training text review classification model; Evaluate a plurality of the text review solutions and determine a text review solution that can be used online.

3. The text review method according to claim 2, characterized in that: The evaluating of the plurality of text review schemes to determine the text review scheme that can be put into use online includes: Construct a full-cycle validation dataset; The full-cycle verification data set is used to evaluate multiple text review solutions to determine the text review solution that can be used online.

4. The text review method according to claim 3, characterized in that: The construction of the full-cycle verification data set includes: Obtain the original training data set, the original validation data set, and the newly added training data; Using the original training data set and the newly added training data to update the original verification data set, to obtain an updated verification data set; The full-cycle validation dataset is constructed according to the original validation dataset and the updated validation dataset.

5. The text review method according to claim 3, characterized in that: The use of the full-cycle verification data set to evaluate the plurality of text review solutions to determine the text review solution that can be put into use includes: Using the pre-trained model to perform feature extraction on the data of the full-cycle verification data set and the training data set, performing retrieval and matching based on the first feature extraction result of the full-cycle verification data set and the first feature extraction result of the training data set, to obtain a first classification result of the full-cycle verification data set; Using the text review classification model to perform feature extraction on the data of the full-cycle verification data set and the training data set, performing retrieval matching based on the second feature extraction result of the full-cycle verification data set and the second feature extraction result of the training data set, to obtain a second classification result of the full-cycle verification data set; Using the text review classification model to classify the data of the full-cycle verification data set, to obtain a third classification result of the full-cycle verification data set; A text review scheme that can be used online is determined based on the first classification result, the second classification result and the third classification result of the full-cycle verification data set.

6. The text review method according to any one of claims 2 to 5, characterized in that: The text review method also includes: Determining whether to trigger an update strategy of the text review classification model, wherein the update strategy includes at least one of a periodic update strategy and a transactional update strategy; When an update strategy of a text review classification model is triggered, the text review classification model is updated.

7. The text review method according to claim 1, characterized in that: The currently used text review scheme is a retrieval scheme based on a pre-trained model or a text review classification model. The initial text review result is optimized according to the review optimization strategy corresponding to the currently used text review scheme to obtain the final text review result, including: Obtaining a plurality of search results having a similarity greater than a preset threshold value among the search results obtained by using the search solution implemented based on the pre-trained model or the text review classification model; Counting the text review categories corresponding to multiple search results that are higher than a preset similarity threshold; The final text review results are determined based on the statistical results.

8. The text review method according to claim 1, characterized in that: The currently used text review scheme is a classification scheme based on a text review classification model. The initial text review result is optimized according to the review optimization strategy corresponding to the currently used text review scheme to obtain the final text review result, including: Determine whether the category probability corresponding to the text review result obtained by the classification scheme implemented based on the text review classification model is less than a preset probability threshold; When the category probability is less than a preset probability threshold, obtaining a search result obtained by a search scheme implemented based on a pre-trained model and a text review classification model; The text classification results output by the text review classification model and the retrieval results obtained by the retrieval scheme implemented based on the pre-trained model and the text review classification model are integrated to determine the final text review result.

9. A text review device, characterized in that: The text review device comprises: A first acquisition unit, used to acquire the text to be reviewed; A review unit, configured to review the text to be reviewed using a currently used text review scheme to obtain an initial text review result, wherein the currently used text review scheme is determined after evaluating a plurality of text review schemes, wherein the plurality of text review schemes include a retrieval scheme and a classification scheme; The optimization unit is used to optimize the initial text review result according to the review optimization strategy corresponding to the currently used text review solution to obtain the final text review result.

10. An electronic device comprising: processor; as well as A memory arranged to store computer executable instructions, which when executed cause the processor to perform the text review method of any one of claims 1 to 8.

11. A computer program product, comprising a computer program or instructions, wherein when the computer program or instructions are executed by a processor, the text review method of any one of claims 1 to 8 is implemented.

Images