Text review method and device, computer readable storage medium and electronic equipment

Abstract

The invention provides a text review method and device, a computer readable storage medium and electronic equipment. The method comprises the following steps: obtaining to-be-examined contents, and determining an initial examination result; the to-be-reviewed content and the initial review result serve as keywords to be retrieved in a feature model, a first retrieval result and a second retrieval result are obtained, the to-be-reviewed content and the initial review result serve as keywords to be retrieved in a search engine, and a third retrieval result and a fourth retrieval result are obtained, each of the feature model and the search engine comprises historical review content and a corresponding historical review result; and determining a total retrieval result according to the first retrieval result, the second retrieval result, the third retrieval result and the fourth retrieval result, and outputting a review result according to the total retrieval result to complete text review of the to-be-reviewed content. Through the method and the device, the problem of low accuracy of a generation result of a language model in the field of text review in the prior art is solved.

Description

Technical Field

[0001] This application relates to the technical field of large language models, and more specifically, to a text review method, apparatus, computer-readable storage medium, and electronic device. Background Art

[0002] Text review is a process of conducting compliance and legality reviews on multiple important elements in a text. Taking the review of contract texts as an example, contract review generally requires the participation of professional legal personnel and is a time-consuming and laborious process. The method of using large model technology to assist in the review not only saves the labor cost of legal personnel but also supplements the risk points that are difficult to discover.

[0003] There are also many applications of large model technology and several methods for document review in the current market. In the field of large model technology, a method, system, and apparatus for enhancing the retrieval ability of an external knowledge base of a large model are disclosed in the prior art. It belongs to the technical field of neural network model applications. By enhancing the retrieval ability of the external knowledge base, the large model can obtain accurate text, thereby making its performance more superior in specific tasks (SQL generation); another example of the prior art proposes an anti-fraud method, apparatus, computer device, and storage medium based on the ChatGPT model. This method uses a labeled rumor dataset to fine-tune a pre-trained model; constructs a chatbot with the generative language model ChatGPT as the upper model and the discriminative language model Bert as the lower model; combines the discriminative language model Bert and the generative language model ChatGPT to crush rumor information in social data. The intuitive chat experience is strong, and the content security review performance is strong.

[0004] The basic method of the large model in the field of text review is as shown in a prior art. The review document is split into sentences to form multiple single sentences, and each single sentence is input into the system database for matching to obtain the sentence similarity of each single sentence; it is determined whether each sentence similarity is greater than a set threshold. When it is greater than the set threshold, the single sentence is marked as a possible risk point. This is also the basic idea of retrieval-based text review. In another prior art with a similar idea, the contract to be reviewed is divided into multiple sub-section texts, and the text similarity score between the sub-section text and the clause text in the contract sample is calculated. The risk prompt information corresponding to the clause text with the highest text similarity score is used as the risk prompt information for the sub-section text.

[0005] However, none of the above prior arts apply the large model in the field of text review and do not deeply combine the characteristics of different application layers of the large model. The generated content is uncontrollable to a certain extent and has a certain "hallucination" problem. The output results are often unsatisfactory. Therefore, the accuracy and business practicality of the large model in the field of text review still need to be improved. Summary of the Invention

[0006] The main objective of this application is to provide a text review method, device, computer-readable storage medium, and electronic device, so as to at least solve the problem of low accuracy of the generation results of language models in the field of text review in the prior art.

[0007] To achieve the above objective, according to one aspect of this application, a text review method is provided, including: obtaining the content to be reviewed and determining an initial review result, where the content to be reviewed includes at least the text information to be reviewed, and the initial review result represents the result after a preliminary review of the content to be reviewed; respectively using the content to be reviewed and the initial review result as keywords to perform searches in a feature model to obtain a first search result and a second search result, and respectively using the content to be reviewed and the initial review result as keywords to perform searches in a search engine to obtain a third search result and a fourth search result, where both the feature model and the search engine include historical review content and corresponding historical review results; determining an overall search result based on the first search result, the second search result, the third search result, and the fourth search result, and outputting a review result based on the overall search result to complete the text review of the content to be reviewed.

[0008] Optionally, determining the initial review result includes: obtaining a target language model, where the target language model is trained using multiple sets of data, and each set of data in the multiple sets of data includes the historical review content and the historical review result; obtaining a prompt, and inputting the prompt and the content to be reviewed into the target language model to analyze the content to be reviewed through the target language model to obtain the initial review result.

[0009] Optionally, the content to be reviewed and the initial review result are respectively used as keywords to be retrieved in the feature model, and a first retrieval result and a second retrieval result are obtained, including: extracting first content features from the historical review content by using the feature model, and extracting first result features from the historical review result by using the feature model; inputting the content to be reviewed into the feature model to extract second content features from the content to be reviewed by using the feature model; inputting the initial review result into the feature model to extract second result features from the initial review result by using the feature model; calculating the similarity between the first content features and the second content features to obtain a first similarity, and arranging the first similarity in a preset order to obtain the first retrieval result corresponding to the content to be reviewed; calculating the similarity between the first result features and the second result features to obtain a second similarity, and arranging the second similarity in the preset order to obtain the second retrieval result corresponding to the initial review result.

[0010] Optionally, the content to be reviewed and the initial review result are respectively used as keywords to be retrieved in a search engine, and a third retrieval result and a fourth retrieval result are obtained, including: calculating the text similarity between the historical review content and the content to be reviewed through an inverted index to obtain a third similarity; calculating the text similarity between the historical review result and the initial review result through the inverted index to obtain a fourth similarity; arranging the third similarity in a preset order to obtain the third retrieval result corresponding to the content to be reviewed; arranging the fourth similarity in the preset order to obtain the fourth retrieval result corresponding to the content to be reviewed.

[0011] Optionally, a total retrieval result is determined according to the first retrieval result, the second retrieval result, the third retrieval result, and the fourth retrieval result, including: obtaining the weight corresponding to each retrieval result and the similarity corresponding to each retrieval result, where the retrieval results include the first retrieval result, the second retrieval result, the third retrieval result, and the fourth retrieval result; calculating the product of the weight and the similarity corresponding to each retrieval result to obtain the total similarity corresponding to each retrieval result, and determining the total retrieval result according to the total similarity.

[0012] Optionally, determining the total retrieval result according to the total similarity includes: combining the first retrieval result, the second retrieval result, the third retrieval result, and the fourth retrieval result to obtain a total retrieval item; in the case where there are identical retrieval items in the total retrieval item, taking the highest total similarity as the total similarity of the retrieval item; sorting the total retrieval items in descending order according to the total similarity, and determining the total retrieval items corresponding to the top preset threshold total similarities as the total retrieval result.

[0013] Optionally, outputting a review result according to the total retrieval result includes: obtaining a prompt word, and inputting the prompt word, the content to be reviewed, and the total retrieval result into a target language model to use the target language model to review the content to be reviewed again and output the review result.

[0014] According to another aspect of the present application, there is provided a text review device, including: a determination unit configured to obtain the content to be reviewed and determine an initial review result, where the content to be reviewed at least includes text information to be reviewed, and the initial review result represents the result after a preliminary review of the content to be reviewed; a retrieval unit configured to respectively use the content to be reviewed and the initial review result as keywords to perform retrieval in a feature model to obtain a first retrieval result and a second retrieval result, and respectively use the content to be reviewed and the initial review result as keywords to perform retrieval in a search engine to obtain a third retrieval result and a fourth retrieval result, where both the feature model and the search engine include historical review content and corresponding historical review results; an output unit configured to determine a total retrieval result according to the first retrieval result, the second retrieval result, the third retrieval result, and the fourth retrieval result, and output a review result according to the total retrieval result to complete the text review of the content to be reviewed.

[0015] According to still another aspect of the present application, there is provided a computer-readable storage medium, where the computer-readable storage medium includes a stored program, and when the program runs, it controls a device where the computer-readable storage medium is located to execute any one of the text review methods.

[0016] According to yet another aspect of the present application, there is provided an electronic device, including: one or more processors, a memory, and one or more programs, where the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include those for executing any one of the text review methods.

[0017] Applying the technical solution of the present application, obtaining the content to be reviewed, and determining the initial review result. Respectively, using the content to be reviewed and the initial review result as keywords to perform retrieval in the feature model, obtaining the first retrieval result and the second retrieval result. Respectively, using the content to be reviewed and the initial review result as keywords to perform retrieval in the search engine, obtaining the third retrieval result and the fourth retrieval result. Determining the total retrieval result according to the first retrieval result, the second retrieval result, the third retrieval result and the fourth retrieval result, and outputting the review result according to the total retrieval result to complete the text review of the content to be reviewed. Compared with the prior art where large models are not applied in the field of text review and the generated content is uncontrollable to a certain extent, there is a certain "hallucination" problem. The present application performs retrieval in the feature model and search engine respectively on the content to be reviewed and the initial review result, and then re-outputs the review result based on the above multiple retrieval results to ensure the controllability and accuracy of the finally output review result. Therefore, it can solve the problem of low accuracy of the results generated by language models in the field of text review in the prior art, and achieve the effect of improving the accuracy of language models. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings forming a part of this application are used to provide a further understanding of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation to the present application. In the drawings:

[0019] Figure 1 It shows a hardware structure block diagram of a mobile terminal for performing a text review method provided by an embodiment of the present application;

[0020] Figure 2 It shows a flowchart of a text review method provided by an embodiment of the present application;

[0021] Figure 3 It shows a flowchart of a specific text review method provided by an embodiment of the present application;

[0022] Figure 4 It shows a flowchart of a multi-dimensional and multi-representation retrieval provided by an embodiment of the present application;

[0023] Figure 5 It shows a flowchart of another multi-dimensional and multi-representation retrieval provided by an embodiment of the present application;

[0024] Figure 6 It shows a structure block diagram of a text review device provided by an embodiment of the present application.

[0025] Among them, the above-mentioned accompanying drawings include the following reference numerals:

[0026] 102. Processor; 104. Memory; 106. Transmission device; 108. Input / output device. Detailed implementation manners

[0027] It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments may be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0028] In order to enable those skilled in the art to better understand the solution of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0029] It should be noted that the terms "first", "second", etc. in the specification, claims and the above-mentioned drawings of the present application are used to distinguish similar objects, and do not have to be used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances so as to describe the embodiments of the present application here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0030] For the convenience of description, some nouns or terms related to the embodiments of the present application are described below:

[0031] Large model: A type of large language generation model represented by ChatGPT, which can be used in fields such as content generation.

[0032] Retrieval enhancement type: Enhancing the generation ability of the large model through the retrieval method to reduce the hallucination problem existing in the large model.

[0033] As introduced in the background technology, the accuracy of the generation results of language models in the field of text review is low in the prior art. To solve the problem of low accuracy of the generation results of language models in the field of text review, the embodiments of the present application provide a text review method, device, computer-readable storage medium and electronic device.

[0034] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0035] The method embodiments provided in the embodiments of the present application can be executed on a mobile terminal, a computer terminal, or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 is a hardware structure block diagram of a mobile terminal for a text review method according to an embodiment of the present invention. As Figure 1 shown, the mobile terminal may include one or more ( Figure 1 only one is shown in the figure) processors 102 (the processor 102 may include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA), and a memory 104 for storing data. Among them, the above mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those of ordinary skill in the art can understand that Figure 1 the structure shown is only schematic and does not limit the structure of the above mobile terminal. For example, the mobile terminal may further include more or fewer components than Figure 1 shown in the figure, or have a different configuration from Figure 1 shown in the figure.

[0036] The memory 104 can be used to store computer programs. For example, software programs and modules of application software, such as the computer program corresponding to the text review method in the embodiments of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, the above method is implemented. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some instances, the memory 104 may further include a memory remotely disposed relative to the processor 102, and these remote memories may be connected to the mobile terminal through a network. Examples of the above network include, but are not limited to, the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof. The transmission device 106 is used to receive or send data via a network. Specific examples of the above network may include a wireless network provided by a communication provider of the mobile terminal. In one instance, the transmission device 106 includes a network adapter (Network Interface Controller, abbreviated as NIC), which can be connected to other network devices through a base station and thus communicate with the Internet. In one instance, the transmission device 106 may be a radio frequency (Radio Frequency, abbreviated as RF) module, which is used to communicate with the Internet wirelessly.

[0037] In this embodiment, a text review method running on a mobile terminal, a computer terminal, or a similar computing device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. And although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0038] Figure 2 is a flowchart of the text review method according to an embodiment of the present application. As Figure 2 shown, the method includes the following steps:

[0039] Step S201, obtain the content to be reviewed and determine an initial review result, where the content to be reviewed at least includes the text information to be reviewed, and the initial review result represents the result after a preliminary review of the content to be reviewed;

[0040] Specifically, the system needs to obtain the text information to be reviewed, that is, the content to be reviewed, which can be a certain clause in a contract, a certain paragraph in a declaration material, or a certain part of an advertisement copy, etc. Subsequently, the large model is used to conduct a preliminary review of this part of the text to obtain an initial review result (R0). In this preliminary review stage, the generation ability of the large model is utilized, but its result may be affected by the "hallucination" problem, that is, the output result of the large model may exceed the scope of its training data or contain information not verified by the training data. Therefore, further enhancement and correction are required.

[0041] Step S202, respectively use the content to be reviewed and the initial review result as keywords to perform searches in the feature model to obtain a first search result and a second search result, and respectively use the content to be reviewed and the initial review result as keywords to perform searches in the search engine to obtain a third search result and a fourth search result, where both the feature model and the search engine include historical review content and corresponding historical review results;

[0042] Specifically, the system will perform retrievals on the content to be reviewed (Query) and the initial review result (R0) separately to find the most relevant review cases for them. First, a retrieval is conducted in the feature model, which is constructed based on embedding technology and can capture the deep semantics of the text. Here, the system will map Query and R0 into this model and perform feature space matching with the historical review content and its corresponding historical review results to obtain the first retrieval result most similar to Query and the second retrieval result most relevant to R0 respectively. At the same time, in the search engine, Query and R0 are also used as keywords to perform a retrieval of the text appearance, obtaining the third retrieval result and the fourth retrieval result. The search engine here can use technologies such as ElasticSearch and measure the similarity between texts through algorithms such as BM25 to ensure that the retrieval results are not only semantically close but also highly relevant on the text surface.

[0043] Step S203, determine the total retrieval result according to the first retrieval result, the second retrieval result, the third retrieval result, and the fourth retrieval result, and output a review result according to the total retrieval result to complete the text review of the content to be reviewed.

[0044] Specifically, the system will comprehensively consider all the retrieval results obtained from the feature model and the search engine and determine a total retrieval result through a certain strategy. This total retrieval result integrates information from two dimensions: semantic depth and text appearance, and more comprehensively reflects the review cases related to the content to be reviewed. Based on this total retrieval result, the large model will generate a review opinion again. This time, the generation process is constrained by historical review cases, significantly improving the accuracy and reliability of the review opinion output by the model, reducing the occurrence of the "hallucination" problem, and thus enhancing the efficiency and quality of text review. In this way, by introducing a retrieval-enhanced large model and combining multi-dimensional retrievals of the feature model and the search engine, the efficiency, accuracy, and practicality of text review by the large language model are effectively improved.

[0045] Through this embodiment, the content to be reviewed is obtained, and the initial review result is determined. The content to be reviewed and the initial review result are respectively used as keywords to perform a search in the feature model, obtaining the first search result and the second search result. The content to be reviewed and the initial review result are respectively used as keywords to perform a search in the search engine, obtaining the third search result and the fourth search result. The total search result is determined based on the first search result, the second search result, the third search result, and the fourth search result, and the review result is output according to the total search result to complete the text review of the content to be reviewed. Compared with the prior art where the large model is not applied in the field of text review and the generated content is uncontrollable to a certain extent and there is a certain "hallucination" problem, in this application, the content to be reviewed and the initial review result are respectively searched in the feature model and the search engine, and then the review result is re-output based on the above multiple search results to ensure the controllability and accuracy of the finally output review result. Therefore, it can solve the problem of low accuracy of the generated results of the language model in the field of text review in the prior art and achieve the effect of improving the accuracy of the language model.

[0046] In the specific implementation process, the above step S201 can be implemented through the following steps: Step S2011: Obtain a target language model, where the target language model is trained using multiple sets of data, and each set of data in the multiple sets of data includes the historical review content and the historical review result; Step S2012: Obtain a prompt, and input the prompt and the content to be reviewed into the target language model to analyze the content to be reviewed through the target language model, obtaining the initial review result. This method generates an initial review result through the target language model, which facilitates further correction of the accuracy of the model output based on the initial review result, so that a more accurate review result is output.

[0047] Specifically, first obtain a "target language model", such as the open-source large model ChatGLM3-6B, which is obtained after initial training with multiple sets of historical review data. The historical review data includes historical review content (i.e., the text reviewed in the past) and the corresponding historical review results (i.e., the review opinions and conclusions for these texts). The training process of the target language model is essentially learning the language patterns and review logics in historical review cases so that the model can understand and adapt to the review requirements in specific fields, such as contract legal review, compliance inspection of declaration materials, etc. Through methods such as fine-tuning or P-tuning v2, which add learnable parameters at multiple layers, a better balance is achieved in terms of parameter efficiency and performance. The model can learn the correlation between the review content and the review results, so that when encountering new content to be reviewed, it can generate preliminary review results based on the learned patterns. After obtaining the target language model, the next step is to obtain "prompt words". Prompt words are the key information to guide the model to understand and generate review opinions, and usually include the review objectives, review scopes, review criteria, etc. Subsequently, input the above prompt words and the content to be reviewed into the target language model. The prompt words play a guiding role here, helping the model understand the review context and requirements. The model will generate a preliminary review opinion, that is, the "initial review result", based on the input content to be reviewed and the prompt words.

[0048] According to the requirements of the open-source large model, organize our existing data into the following format: {"conversations": [{"role": "user", "content": "Failure by any party to perform any term of this Agreement shall be deemed a breach of contract. The breaching party shall bear the direct economic losses caused to the non-breaching party due to its own breach of contract."}, {"role": "assistant", "content": "Please verify and modify the loss compensation clause. Article X of the submitted text limits the scope of loss compensation for liability for breach of contract to direct losses / limits the upper limit of liability for breach of contract, etc. Please consider whether direct losses / liability limits can cover the scope of \"foreseeable losses\" stipulated in Article 584 of the Civil Code."}]}). This data format still maintains the dialogue ability of the large model. Through adjusting parameters such as the learning rate and batch size, multiple rounds of iteration are carried out to complete the entire process of fine-tuning the large model. The fine-tuned large model already has professional domain knowledge and has a certain adaptability to common content to be reviewed. Prompt engineering will be added here to drive the large model to give its preliminary review opinions on the text content. For example, when reviewing, the system gives the following prompt words: "You will act as a text review assistant and review the following content: \"{content}\", requirements: give accurate and detailed review opinions and answer in Chinese." The large model will give the preliminary review opinion R0 according to the requirements. However, the above steps only rely on the ability of the fine-tuned large model and still cannot solve the problem of large model hallucinations. For example, given content: "Failure by any party to perform any term of this Agreement shall be deemed a breach of contract. The breaching party shall bear the direct economic losses caused to the non-breaching party due to its own breach of contract.", its output result is: "When using similar terms, it is best to ensure the clarity and distinctness of the text to prevent subsequent interpretation disputes. Consider adding definitions for some terms or clarifying the rights and obligations of each party when necessary to strengthen the enforceability and comprehensibility of the contract." It can be seen that this result does not really focus on the business key points such as direct losses and compensation clauses.

[0049] In some alternative embodiments, the above step S202 can be implemented through the following steps: Step S2021: Use the feature model to extract features from the historical review content to obtain the first content feature, and use the feature model to extract features from the historical review result to obtain the first result feature; Step S2022: Input the content to be reviewed into the feature model to use the feature model to extract features from the content to be reviewed to obtain the second content feature; Step S2023: Input the initial review result into the feature model to use the feature model to extract features from the initial review result to obtain the second result feature; Step S2024: Calculate the similarity between the first content feature and the second content feature to obtain the first similarity, and arrange the first similarity in a preset order to obtain the first retrieval result corresponding to the content to be reviewed; Step S2025: Calculate the similarity between the first result feature and the second result feature to obtain the second similarity, and arrange the second similarity in the preset order to obtain the second retrieval result corresponding to the initial review result. This method performs searches in different dimensions and different search methods through the above steps, so as to perform multi-dimensional and multi-representation retrieval enhancement, enabling the text review process to simultaneously refer to semantic similarity and text appearance consistency, and providing a more comprehensive and reliable review basis.

[0050] Specifically, use the feature model to extract features from the historical review content and the historical review result to obtain two sets of feature vectors: the first content feature and the first result feature. The feature model is used to convert text into vectors of a fixed dimension, and these vectors can capture the semantic features of the text. Through feature extraction, the historical review data is converted into feature vectors, providing a necessary basis for subsequent similarity calculation. The feature model used is the BAAI / bge-large-zh model, which supports Chinese and English embeddings, supports an input feature length of 512, and has an output feature dimension of 1024 dimensions. Since this model already has strong feature embedding capabilities, and it is also difficult for the review data set to provide positive and negative sample pairs, the feature model will not be fine-tuned. Taking the retrieval of the content to be reviewed Query as an example, features will be extracted from the Query, and at the same time, features will be extracted from the review content in the review data set. The features of the Query will be compared with the features of all review contents, and the cosine similarity is used for similarity calculation. After calculating the similarity, it is sorted from high to low to obtain the retrieval result of the Query. Input the content to be reviewed into the feature model to obtain the second content feature. The feature embedding retrieval of the large model review result R0 is carried out in the review opinions of the review data set, and its method is consistent with the retrieval method of the Query to obtain the retrieval result of R0. That is, similarly, input the initial review result into the feature model to obtain the second result feature.

[0051] Subsequently, by calculating the similarity between the first content feature of the historical data and the second content feature of the content to be reviewed, a first similarity is obtained. In addition to the cosine similarity, methods such as Euclidean distance can also be used for the similarity. Here, the first similarity reflects the semantic proximity between the content to be reviewed and the historical review content. Similarly, by calculating the similarity between the first result feature of the historical data and the second result feature of the initial review result, a second similarity is obtained, which reflects the semantic similarity between the initial review result and the historical review result. The calculated first similarity and second similarity are respectively arranged in a preset order. The preset order usually means sorting the similarities from high to low, so as to obtain the first retrieval result most relevant to the content to be reviewed and the second retrieval result most relevant to the initial review result. These retrieval results are sorted based on the semantic features extracted by the feature model. Therefore, they can provide historical review cases that are semantically closest to the content to be reviewed or the initial review result, providing a more accurate reference for subsequent reviews.

[0052] In some alternative embodiments, the above step S202 can also be implemented through the following steps: Step S2026: Calculate the text similarity between the historical review content and the content to be reviewed through an inverted index to obtain a third similarity; Step S2027: Calculate the text similarity between the historical review result and the initial review result through the inverted index to obtain a fourth similarity; Step S2028: Arrange the third similarity in a preset order to obtain the third retrieval result corresponding to the content to be reviewed; Step S2029: Arrange the fourth similarity in the preset order to obtain the fourth retrieval result corresponding to the content to be reviewed. The text similarity calculated by this method through the inverted index ensures that the retrieval results are not only semantically close but also highly consistent in terms of text appearance (i.e., the literal form and structure of the text), which is crucial for ensuring the pertinence and accuracy of the review results.

[0053] Specifically, the ElasticSearch search engine is used for retrieval. ElasticSearch uses the BM25 algorithm for text. First, an index is established for the data in the review data group, and the similarity between texts is calculated through the inverted index. This method ensures text appearance consistency and uses the ES engine for calculation. Taking the retrieval of the content to be reviewed Query as an example, the review content of the review data group is put into the ES engine in advance, and Query is used as the query condition to query in ES, and the results are sorted according to the retrieval scores after the query. The review result R0 of the large model is retrieved by ES in the same way.

[0054] Using an inverted index, text similarity calculations are performed on historical review content and content to be reviewed. Inverted indexes are typically constructed based on algorithms such as term frequency, inverse document frequency (TF-IDF), or BM25. These algorithms can quantify the importance of a term in a document, thereby measuring the similarity between two texts. The calculated third similarity reflects the similarity of the historical review content and the current content to be reviewed in terms of the apparent text. Similarly, the inverted index is used to calculate the text similarity between the historical review results and the preliminary review results (initial review results) generated by the large model. The obtained fourth similarity indicates the proximity of the historical review results and the preliminary review opinions at the text level, which is very important for calibrating and supplementing the preliminary review opinions. Subsequently, the calculated third similarities are arranged in a preset order (usually from high to low), and the resulting list of retrieval results is the third retrieval result corresponding to the content to be reviewed. This list contains the historical review content that is most similar to the content to be reviewed in terms of text similarity, providing an important reference for subsequent reviews. Similarly, the system sorts the fourth similarities, and the resulting list is the fourth retrieval result corresponding to the initial review results. This list contains the historical review results that are most similar to the preliminary review opinions in terms of text similarity, helping to correct and optimize the preliminary review opinions to ensure their accuracy and professionalism.

[0055] In some alternative embodiments, step S203 described above can also be implemented through the following steps: Step S2031: Obtain the weight corresponding to each of the retrieval results and obtain the similarity corresponding to each of the retrieval results, where the retrieval results include the first retrieval result, the second retrieval result, the third retrieval result, and the fourth retrieval result; Step S2032: Calculate the product of the weight and the similarity corresponding to each of the retrieval results to obtain the total similarity corresponding to each of the retrieval results, and determine the total retrieval result based on the total similarity. By introducing weights, this method can perform weighted processing according to the different importance of the retrieval results, ensuring that the final output total retrieval result is more valuable as a reference.

[0056] In the specific implementation process, first, four retrieval results related to the content to be reviewed are obtained: the first retrieval result (the retrieval result of the content to be reviewed based on the feature model), the second retrieval result (the retrieval result of the initial review result based on the feature model), the third retrieval result (the text similarity retrieval result of the content to be reviewed based on the inverted index), and the fourth retrieval result (the text similarity retrieval result of the initial review result based on the inverted index). Each retrieval result will be assigned a weight, which reflects the importance of the retrieval result in the final decision. For example, if the retrieval result of the feature model is considered to be more accurate in semantic understanding, it may be assigned a higher weight; while the retrieval result of the inverted index may be more excellent in text appearance consistency and will also be assigned a corresponding weight according to its importance. The four retrievals obtained are represented by the following sorted list: {Query feature retrieval result, R0 feature retrieval result, Query ES retrieval result, R0 ES retrieval result}. The product of the weight and the similarity is used as the final value of each result item. Specifically: (1) Normalize the similarities in the four sorted lists respectively to ensure that their similarities are in the same dimension; (2) Take the top TopK values from each list, and multiply the weight assigned to the list by the similarity as the similarity of each item in the list; The weights assigned to each method are {w1w2, (1 - w1)w2, w1(1 - w2), (1 - w1)(1 - w2)}, where w1 is the weight of Query in multiple dimensions and w2 is the weight of the feature retrieval method in multiple representations. After multiple experiments, it can be seen that the effect of Query is better than that of R0, and the feature retrieval is better than the ES retrieval. Therefore, w1 = 0.6 and w2 = 0.7 are set; (3) Merge each item in the four lists. If the same item appears multiple times, take the highest similarity score of the item as the similarity of the item; (4) Sort the merged list in descending order of similarity as the final sorted list.

[0057] Next, the system will calculate the total similarity of each retrieval result. This step multiplies the similarity of the retrieval result by its corresponding weight. For the first retrieval result, its total similarity will be the product of the first content feature similarity and its weight; for the second retrieval result, its total similarity is the product of the first result feature similarity and its weight; The total similarities of the third and fourth retrieval results are the products of the text similarities calculated based on the inverted index and their respective weights, and the weight values are as shown in the above list. After calculating the total similarities of all retrieval results, the retrieval results are sorted according to these total similarities, and a total retrieval result is determined in descending order. This total retrieval result is the result of comprehensively considering the retrieval results in different dimensions and representations, reflecting the historical review cases that are most relevant to the content to be reviewed and the initial review opinions in terms of semantics and text appearance.

[0058] In some alternative embodiments, the above step S2032 can be implemented through the following steps: Step S2033: Combine the first search result, the second search result, the third search result, and the fourth search result to obtain a total search item; Step S2034: In the case where there are identical search items in the total search item, use the highest total similarity as the total similarity of the search item; Step S2035: Sort the total search items in descending order of the total similarity, and determine the first preset threshold number of total search items corresponding to the total similarity as the total search result. This method processes duplicate search items through the above steps to ensure that each case in the output total search result is the most representative and relevant, avoiding multiple references to the same case, and improving the review efficiency and the clarity of information.

[0059] In the specific implementation process, for the total search result corresponding to the total similarity calculated through the above steps, the first search result (search result of the content to be reviewed based on the feature model), the second search result (search result of the initial review result based on the feature model), the third search result (text similarity search result of the content to be reviewed based on the inverted index), and the fourth search result (text similarity search result of the initial review result based on the inverted index) are combined. These search results may include multiple documents or cases related to the content to be reviewed. Through combination, a set of total search items containing all possible relevant items is formed. In the combined total search items, there may be cases where some historical review contents or results are retrieved multiple times, that is, duplicate search items. To ensure that each historical case is only considered once and appears in the most relevant form, these duplicate items are checked, and the version with the highest total similarity among them is used as the final representation of the duplicate item. The total similarity is obtained by multiplying the original similarity of the search result by its preset weight, which can ensure that even if a case is found through multiple search methods, the system will retain the version that is closest to the content to be reviewed in different search strategies, avoiding information redundancy and improving the accuracy of the search. Finally, the combined and deduplicated total search items are sorted according to their total similarity, from high to low. After sorting, the system usually selects the first preset threshold number of search items as the final total search result. This preset threshold can be a number preset according to the review requirements and efficiency requirements, such as selecting the top 10 or 20 most relevant search items. In this way, the system can output the cases most relevant to the content to be reviewed and the preliminary review result, providing the most valuable reference information for the next review.

[0060] In some alternative embodiments, the above step S203 further includes the following steps: Step S2036: Obtain a prompt, and input the prompt, the content to be reviewed, and the total retrieval result into a target language model to review the content to be reviewed again using the target language model and output the review result. Through the above steps, the method conducts review and output through the target language model once again, and combines the above retrieval result to output a more accurate review result.

[0061] Specifically, based on the prompt, the content to be reviewed and the previously determined total retrieval result (i.e., the top preset threshold number of historical review cases most relevant to the content to be reviewed) are combined to form a data packet input to the target language model. Through re-review by the target language model, a review result is output. The model will conduct in-depth analysis and review of the text again according to the input prompt, the content to be reviewed, and its related total retrieval result, and generate a final review opinion. This process makes full use of the guiding role of historical review data, and combines the generation ability and understanding ability of the model to ensure that the output review result is the most optimized review opinion based on existing knowledge and historical cases, with high accuracy and professionalism. Specifically, the following prompt can be input to the large model: "You will act as a text review assistant and review the following content: "{content}". The review result needs to refer to the following existing review cases: 1. When the review content is {q1}, the review result is {r1}; 2. When the review content is {q2}, the review result is {r2};...; k. When the review content is {qk}, the review result is {rk}; Requirements: The review result needs to be accurate and detailed. If there is no relevant information in the provided review cases, please try to answer by yourself, but tell the user that your answer does not refer to any relevant context and answer in Chinese." For the above prompt, qk and rk are respectively the kth reference review content and review opinion in the retrieval result. In this way, by utilizing the generation ability of the fine-tuned large model and the information of similar reference review opinions, a relatively accurate review opinion can be obtained, thus completing the two-stage large model text review process.

[0062] In order to enable those skilled in the art to more clearly understand the technical solution of the present application, the implementation process of the text review method of the present application will be described in detail below in conjunction with specific embodiments.

[0063] This embodiment relates to a specific text review method, as Figure 3 shown, including the following steps:

[0064] Step S1: Start;

[0065] Step S2: Pair (review content, review opinion), input into the large model for fine-tuning;

[0066] Step S3: The content to be reviewed Query + prompt engineering, and use the fine-tuned large model for review to obtain the large model review result R0;

[0067] Step S4: The content to be reviewed Query and the large model review result R0 are subjected to multi-dimensional multi-representation retrieval, including feature embedding retrieval and ES retrieval. A schematic diagram of the process of multi-dimensional multi-representation retrieval is as Figure 4 shown. The query content is searched for relevance in the vector database, and the relevant content obtained and the query content are then input into the large model together, and the retrieval result is further obtained based on prompt engineering (prompt words);

[0068] Step S5: Obtain the Query retrieval result and the R0 retrieval result respectively;

[0069] Step S6: Re-rank the retrieval results;

[0070] Step S7: The content to be reviewed Query and the retrieval results are input into the fine-tuned large model;

[0071] Step S8: Based on prompt engineering (prompt words);

[0072] Step S9: Obtain the final review opinion;

[0073] Step S10: End.

[0074] Figure 5 A schematic diagram of the process of another multi-dimensional multi-representation retrieval is shown, as Figure 5 shown, including the following steps:

[0075] Step S11: Start;

[0076] Step S12: The content to be reviewed Query is respectively used for feature embedding retrieval and ES retrieval to obtain the Query feature retrieval result and the Query ES retrieval result;

[0077] Step S13: The initial review result R0 of the large model is respectively used for feature embedding retrieval and ES retrieval to obtain the R0 feature retrieval result and the R0 ES retrieval result;

[0078] Step S14: Perform re-ranking;

[0079] Step S15: End.

[0080] The embodiments of the present application also provide a text review device. It should be noted that the text review device in the embodiments of the present application can be used to execute the text review method provided in the embodiments of the present application. The device is used to implement the above embodiments and preferred implementation manners, and those that have been described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that can achieve a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware is also possible and contemplated.

[0081] The following introduces the text review device provided in the embodiments of the present application.

[0082] Figure 6 is a schematic diagram of the text review device according to the embodiments of the present application. As Figure 6 shown, the device includes:

[0083] A determination unit 10, configured to obtain the content to be reviewed and determine an initial review result, where the content to be reviewed includes at least the text information to be reviewed, and the initial review result represents the result after a preliminary review of the content to be reviewed;

[0084] Specifically, the system needs to obtain the text information to be reviewed, that is, the content to be reviewed, which can be a certain clause in a contract, a certain paragraph in a declaration material, or a certain part of an advertisement copy, etc. Subsequently, the large model is used to conduct a preliminary review of this part of the text to obtain an initial review result (R0). This preliminary review stage utilizes the generation ability of the large model, but its result may be affected by the "hallucination" problem, that is, the output result of the large model may exceed the scope of its training data or contain information not verified by the training data. Therefore, further enhancement and correction are required.

[0085] A retrieval unit 20, configured to respectively retrieve the content to be reviewed and the initial review result as keywords in the feature model to obtain a first retrieval result and a second retrieval result, and respectively retrieve the content to be reviewed and the initial review result as keywords in the search engine to obtain a third retrieval result and a fourth retrieval result, where both the feature model and the search engine include historical review content and corresponding historical review results;

[0086] Specifically, the system will perform retrievals on the content to be reviewed (Query) and the initial review result (R0) separately to find the most relevant review cases for them. First, a retrieval is performed in the feature model, which is constructed based on embedding technology and can capture the deep semantics of the text. Here, the system will map Query and R0 into this model and perform feature space matching with the historical review content and its corresponding historical review results to obtain the first retrieval result most similar to Query and the second retrieval result most relevant to R0 respectively. At the same time, in the search engine, using Query and R0 as keywords, a retrieval of the text appearance is also performed to obtain the third retrieval result and the fourth retrieval result. The search engine here can use technologies such as ElasticSearch and measure the similarity between texts through algorithms such as BM25 to ensure that the retrieval results are not only semantically close but also highly relevant on the text surface.

[0087] An output unit 30 is configured to determine a total retrieval result according to the first retrieval result, the second retrieval result, the third retrieval result, and the fourth retrieval result, and output a review result according to the total retrieval result to complete the text review of the content to be reviewed.

[0088] Specifically, the system will comprehensively consider all the retrieval results obtained from the feature model and the search engine and determine a total retrieval result through a certain strategy. This total retrieval result integrates information from two dimensions: semantic depth and text appearance, and more comprehensively reflects the review cases related to the content to be reviewed. Based on this total retrieval result, the large model will generate review opinions again. This time, the generation process is constrained by historical review cases, significantly improving the accuracy and reliability of the review opinions output by the model, reducing the occurrence of the "hallucination" problem, and thus enhancing the efficiency and quality of text review. In this way, by introducing a retrieval-enhanced large model and combining multi-dimensional retrievals of the feature model and the search engine, the efficiency, accuracy, and practicality of text review by the large language model are effectively improved.

[0089] Through this embodiment, the content to be reviewed is obtained, and the initial review result is determined. The content to be reviewed and the initial review result are respectively used as keywords to perform a search in the feature model, obtaining the first search result and the second search result. The content to be reviewed and the initial review result are respectively used as keywords to perform a search in the search engine, obtaining the third search result and the fourth search result. The total search result is determined according to the first search result, the second search result, the third search result, and the fourth search result, and the review result is output according to the total search result to complete the text review of the content to be reviewed. Compared with the prior art where large models are not applied in the field of text review and the generated content is uncontrollable to a certain extent, with a certain "hallucination" problem, in this application, the feature model search and the search engine search are respectively performed on the content to be reviewed and the initial review result, and then the review result is re-output based on the above multiple search results to ensure the controllability and accuracy of the finally output review result. Therefore, it can solve the problem of low accuracy of the results generated by the language model in the field of text review in the prior art and achieve the effect of improving the accuracy of the language model.

[0090] In the specific implementation process, the above determination unit includes a first acquisition module and an analysis module. The first acquisition module is used to acquire the target language model, where the target language model is trained using multiple sets of data, and each set of data in the multiple sets of data includes the historical review content and the historical review result; the analysis module is used to acquire the prompt word, input the prompt word and the content to be reviewed into the target language model, so as to analyze the content to be reviewed through the target language model to obtain the initial review result. This device generates an initial review result through the target language model, which is convenient for further correcting the accuracy of the model output based on the initial review result, so that a more accurate review result is output.

[0091] Specifically, first obtain a "target language model", such as the open-source large model ChatGLM3-6B, which is obtained after initial training with multiple sets of historical review data. The historical review data includes historical review content (i.e., the text reviewed in the past) and the corresponding historical review results (i.e., the review opinions and conclusions for these texts). The training process of the target language model is essentially learning the language patterns and review logics in historical review cases so that the model can understand and adapt to the review requirements in specific fields, such as contract legal review, compliance inspection of declaration materials, etc. Through devices such as fine-tuning or P-tuning v2, which add learnable parameters at multiple layers, a better balance is achieved in terms of parameter efficiency and performance. The model can learn the correlation between review content and review results, so that when encountering new content to be reviewed, it can generate preliminary review results based on the learned patterns. After obtaining the target language model, the next step is to obtain "prompt words". Prompt words are the key information that guides the model to understand and generate review opinions, and usually include the review objectives, review scopes, review criteria, etc. Subsequently, input the above prompt words and the content to be reviewed into the target language model. The prompt words play a guiding role here, helping the model understand the review context and requirements. The model will generate a preliminary review opinion, that is, the "initial review result", based on the input content to be reviewed and the prompt words.

[0092] According to the requirements of the open-source large model, organize our existing data into the following format: {"conversations": [{"role": "user", "content": "Failure by any party to perform any term of this Agreement shall be deemed a breach of contract. The breaching party shall bear the direct economic losses caused to the non-breaching party due to its own breach of contract."}, {"role": "assistant", "content": "Please verify and modify the loss compensation clause. Article X of the submission text limits the scope of loss compensation for liability for breach of contract to direct losses / limits the upper limit of liability for breach of contract, etc. Please consider whether direct losses / liability limits can cover the scope of \"foreseeable losses\" stipulated in Article 584 of the Civil Code."}]} The data format still maintains the dialogue ability of the large model. Through multiple rounds of iteration by adjusting parameters such as the learning rate and batch size, the entire process of fine-tuning the large model is completed. The fine-tuned large model already has professional domain knowledge and has a certain adaptability to common content to be reviewed. Here, prompt engineering will be added to drive the large model to give its preliminary review opinions on the text content. For example, during the review, the system gives the following prompt words: "You will act as a text review assistant and review the following content: \"{content}\", requirements: give accurate and detailed review opinions and answer in Chinese." The large model will give preliminary review opinions R0 according to the requirements. However, the above steps only rely on the capabilities of the fine-tuned large model and still cannot solve the problem of large model hallucinations. For example, given content: "Failure by any party to perform any term of this Agreement shall be deemed a breach of contract. The breaching party shall bear the direct economic losses caused to the non-breaching party due to its own breach of contract.", its output result is: "When using similar terms, it is best to ensure the clarity and explicitness of the text to prevent subsequent interpretation disputes. Consider adding definitions for some terms or clarifying the rights and obligations of each party when necessary to enhance the enforceability and understandability of the contract." It can be seen that this result does not really focus on the business key points such as direct losses and compensation clauses.

[0093] In some alternative embodiments, the above-mentioned retrieval unit includes a first extraction module, a second extraction module, a third extraction module, a fourth extraction module, and an arrangement module. The first extraction module is configured to use the feature model to extract features from the historical review content to obtain first content features, and use the feature model to extract features from the historical review result to obtain first result features. The second extraction module is configured to input the content to be reviewed into the feature model to extract features from the content to be reviewed using the feature model to obtain second content features. The third extraction module is configured to input the initial review result into the feature model to extract features from the initial review result using the feature model to obtain second result features. The fourth extraction module is configured to calculate the similarity between the first content features and the second content features to obtain a first similarity, and arrange the first similarity in a preset order to obtain the first retrieval result corresponding to the content to be reviewed. The arrangement module is configured to calculate the similarity between the first result features and the second result features to obtain a second similarity, and arrange the second similarity in the preset order to obtain the second retrieval result corresponding to the initial review result. The device performs searches in different dimensions and different search methods through the above steps, so as to perform multi-dimensional and multi-representation retrieval enhancement, enabling the text review process to simultaneously refer to semantic similarity and text appearance consistency, and providing a more comprehensive and reliable review basis.

[0094] Specifically, a feature model is used to extract features from historical review content and historical review results, obtaining two sets of feature vectors: the first content feature and the first result feature. The feature model is used to convert text into vectors of a fixed dimension, and these vectors can capture the semantic features of the text. Through feature extraction, historical review data is converted into feature vectors, providing a necessary basis for subsequent similarity calculation. The feature model used is the BAAI / bge-large-zh model, which supports Chinese and English embeddings. This model supports an input feature length of 512 and an output feature dimension of 1024. Since this model already has strong feature embedding capabilities and it is difficult for the review data set to provide positive and negative sample pairs, the feature model will not be fine-tuned. Taking the retrieval of the content to be reviewed Query as an example, features will be extracted from the Query, and at the same time, features will be extracted from the review content in the review data set. The features of the Query will be compared with the features of all review content, and the cosine similarity is used for similarity calculation. After calculating the similarity, the results are sorted from high to low according to the similarity to obtain the retrieval results of the Query. The content to be reviewed is input into the feature model to obtain the second content feature. The feature embedding retrieval of the large model review result R0 is carried out in the review opinions of the review data set, and its method is consistent with the retrieval method of the Query to obtain the retrieval results of R0. That is, similarly, the initial review result is input into the feature model to obtain the second result feature.

[0095] After that, by calculating the similarity between the first content feature of the historical data and the second content feature of the content to be reviewed, the first similarity is obtained. In addition to the cosine similarity, the similarity can also use devices such as Euclidean distance. Here, the first similarity reflects the semantic proximity between the content to be reviewed and the historical review content. Similarly, by calculating the similarity between the first result feature of the historical data and the second result feature of the initial review result, the second similarity is obtained, which reflects the semantic similarity between the initial review result and the historical review result. The calculated first similarity and second similarity are respectively arranged in a preset order. The preset order usually refers to sorting the similarity from high to low, so as to obtain the first retrieval result most relevant to the content to be reviewed and the second retrieval result most relevant to the initial review result. These retrieval results are sorted based on the semantic features extracted by the feature model. Therefore, they can provide historical review cases that are semantically closest to the content to be reviewed or the initial review result, providing a more accurate reference for subsequent reviews.

[0096] In some alternative embodiments, the above-mentioned retrieval unit further includes a first calculation module, a second calculation module, a first sorting module, and a second sorting module. The first calculation module is configured to calculate the text similarity between the historical review content and the content to be reviewed through an inverted index to obtain a third similarity; the second calculation module is configured to calculate the text similarity between the historical review result and the initial review result through the inverted index to obtain a fourth similarity; the first sorting module is configured to sort the third similarity in a preset order to obtain the third retrieval result corresponding to the content to be reviewed; the second sorting module is configured to sort the fourth similarity in the preset order to obtain the fourth retrieval result corresponding to the content to be reviewed. The text similarity calculated by the device through the inverted index ensures that the retrieval results are not only semantically close but also highly consistent in terms of text appearance (i.e., the literal form and structure of the text), which is crucial for ensuring the pertinence and accuracy of the review results.

[0097] Specifically, the ElasticSearch search engine is used for retrieval. ElasticSearch uses the BM25 algorithm for text. First, indexes are established for the data in the review data group, and the similarity between texts is calculated through the inverted index. This method ensures text appearance consistency and uses the ES engine for calculation. Taking the retrieval of the content to be reviewed Query as an example, the review content of the review data group is put into the ES engine in advance, and Query is used as the query condition for querying in ES. After the query, the sorting is performed according to the retrieval score. The review result R0 of the large model is retrieved by ES in the same way.

[0098] Using an inverted index, text similarity calculations are performed on historical review content and content to be reviewed. Inverted indexes are typically constructed based on algorithms such as term frequency, inverse document frequency (TF-IDF), or BM25. These algorithms can quantify the importance of a term in a document, thereby measuring the similarity between two texts. The calculated third similarity reflects the degree of similarity between the historical review content and the current content to be reviewed in terms of the apparent text. Similarly, the inverted index is used to calculate the text similarity between the historical review results and the preliminary review results (initial review results) generated by the large model. The obtained fourth similarity indicates the degree of proximity between the historical review results and the preliminary review opinions at the text level, which is very important for calibrating and supplementing the preliminary review opinions. Subsequently, the calculated third similarities are arranged in a preset order (usually from high to low), and the resulting list of retrieval results is the third retrieval result corresponding to the content to be reviewed. This list contains the historical review content that is most similar to the content to be reviewed in terms of text similarity, providing an important reference for subsequent reviews. Similarly, the system sorts the fourth similarities, and the resulting list is the fourth retrieval result corresponding to the initial review results. This list contains the historical review results that are most similar to the preliminary review opinions in terms of text similarity, helping to correct and optimize the preliminary review opinions to ensure their accuracy and professionalism.

[0099] In some alternative embodiments, the above output unit includes a second acquisition module and a first determination module. The second acquisition module is configured to acquire the weight corresponding to each of the retrieval results and the similarity corresponding to each of the retrieval results, where the retrieval results include the first retrieval result, the second retrieval result, the third retrieval result, and the fourth retrieval result; the first determination module is configured to calculate the product of the weight and the similarity corresponding to each of the retrieval results to obtain the total similarity corresponding to each of the retrieval results, and determine the total retrieval result based on the total similarity. By introducing weights, the device can perform weighted processing according to the different importance of the retrieval results, ensuring that the final output total retrieval result is more valuable as a reference.

[0100] In the specific implementation process, first, four retrieval results related to the content to be reviewed are obtained: the first retrieval result (the retrieval result of the content to be reviewed based on the feature model), the second retrieval result (the retrieval result of the initial review result based on the feature model), the third retrieval result (the text similarity retrieval result of the content to be reviewed based on the inverted index), and the fourth retrieval result (the text similarity retrieval result of the initial review result based on the inverted index). Each retrieval result will be assigned a weight, which reflects the importance of the retrieval result in the final decision. For example, if the retrieval result of the feature model is considered to be more accurate in semantic understanding, it may be assigned a higher weight; while the retrieval result of the inverted index may be more excellent in text apparent consistency and will also be assigned a corresponding weight according to its importance. The four retrievals obtained are represented by the following sorted list: {Query feature retrieval result, R0 feature retrieval result, Query ES retrieval result, R0 ES retrieval result}. Profit Use the product of the weight and the similarity as the final value of each result item . Specifically: (1) Normalize the similarities in the four sorted lists respectively to ensure that their similarities are in the same dimension; (2) Take the top TopK values for each list, and multiply the weight assigned to the list by the similarity as the similarity of each item in the list; The weights assigned to each method are {w1w2, (1 - w1)w2, w1(1 - w2), (1 - w1)(1 - w2)}, where w1 is the weight of Query in multiple dimensions and w2 is the weight of the feature retrieval method in multiple representations. Through multiple experiments, it can be seen that the effect of Query is better than that of R0, and the feature retrieval is better than the ES retrieval. Therefore, set w1 = 0.6 and w2 = 0.7; (3) Merge each item of the four lists. If the same item appears multiple times, take the highest similarity score of the item as the similarity of the item; (4) Sort the merged list in descending order of similarity as the final sorted list.

[0101] Next, the system will calculate the total similarity of each retrieval result. In this step, the similarity of the retrieval result is multiplied by its corresponding weight. For the first retrieval result, its total similarity will be the product of the first content feature similarity and its weight; for the second retrieval result, its total similarity is the product of the first result feature similarity and its weight; The total similarities of the third and fourth retrieval results are the products of the text similarities calculated based on the inverted index and their respective weights, and the weight values are as shown in the above list. After calculating the total similarities of all retrieval results, sort the retrieval results according to these total similarities, and determine a total retrieval result in descending order. This total retrieval result is the result of comprehensively considering the retrieval results in different dimensions and representations, and reflects the historical review cases that are most relevant to the content to be reviewed and the initial review opinions in terms of semantics and text appearance.

[0102] In some alternative embodiments, the above first determination module further includes a merging module, a second determination module, and a third determination module. The merging module is configured to merge the first search result, the second search result, the third search result, and the fourth search result to obtain a total search item. The second determination module is configured to, when there are identical search items in the total search item, use the highest total similarity as the total similarity of the search item. The third determination module is configured to sort the total search items in descending order according to the total similarity, and determine the first preset threshold number of total search items corresponding to the total similarity as the total search result. The apparatus processes duplicate search items through the above steps, ensuring that each case in the output total search result is the most representative and relevant, avoiding multiple references to the same case, and improving the review efficiency and the clarity of information.

[0103] In the specific implementation process, for the total search result corresponding to the total similarity calculated through the above steps, the first search result (the search result of the content to be reviewed based on the feature model), the second search result (the search result of the initial review result based on the feature model), the third search result (the text similarity search result of the content to be reviewed based on the inverted index), and the fourth search result (the text similarity search result of the initial review result based on the inverted index) are merged. These search results may include multiple documents or cases related to the content to be reviewed. Through merging, a set of total search items containing all possible relevant items is formed. In the merged total search items, there may be cases where some historical review contents or results are retrieved multiple times, that is, duplicate search items. To ensure that each historical case is considered only once and appears in the most relevant form, these duplicate items are checked, and the version with the highest total similarity among them is used as the final representation of the duplicate item. The total similarity is obtained by multiplying the original similarity of the search result by its preset weight, which ensures that even if a case is found through multiple search methods, the system will retain the version that is closest to the content to be reviewed in different search strategies, avoiding information redundancy and improving the accuracy of the search. Finally, the merged and deduplicated total search items are sorted according to their total similarity, from high to low. After sorting, the system usually selects the first preset threshold number of search items as the final total search result. This preset threshold can be a number preset according to the review requirements and efficiency requirements, such as selecting the top 10 or 20 most relevant search items. In this way, the system can output the cases most relevant to the content to be reviewed and the preliminary review result, providing the most valuable reference information for the next review.

[0104] In some alternative embodiments, the above output unit further includes an output module, which is configured to obtain a prompt word, input the prompt word, the content to be reviewed, and the total retrieval result into a target language model, so as to use the target language model to review the content to be reviewed again and output the review result. Through the above steps, the device conducts review and output through the target language model once again, and combines the above retrieval results to output a more accurate review result.

[0105] Specifically, based on the prompt word, the content to be reviewed and the previously determined total retrieval result (i.e., the first preset threshold number of historical review cases most relevant to the content to be reviewed) are combined to form a data packet input to the target language model. Through re-review by the target language model, the review result is output. The model will re-analyze and review the text based on the input prompt word, the content to be reviewed, and its relevant total retrieval result, and generate a final review opinion. This process makes full use of the guiding role of historical review data, and combines the generation ability and understanding ability of the model to ensure that the output review result is the most optimized review opinion based on existing knowledge and historical cases, with high accuracy and professionalism. Specifically, the following prompt word can be input to the large model: "You will act as a text review assistant and review the following content: "{content}". The review result needs to refer to the following existing review cases: 1. When the review content is {q1}, the review result is {r1}; 2. When the review content is {q2}, the review result is {r2};...; k. When the review content is {qk}, the review result is {rk}; Requirements: The review result needs to be accurate and detailed. If there is no relevant information in the provided review cases, please try to answer by yourself, but tell the user that your answer does not refer to any relevant context, and answer in Chinese." For the above prompt word, qk and rk are respectively the kth reference review content and review opinion in the retrieval result. In this way, by utilizing the generation ability of the fine-tuned large model and the information of similar reference review opinions, a relatively accurate review opinion can be obtained, thus completing the two-stage large model text review process.

[0106] The text review device includes a processor and a memory. The above determination unit, retrieval unit, output unit, etc. are all stored in the memory as program units, and the processor executes the above program units stored in the memory to implement corresponding functions. The above modules are all located in the same processor; or, the above modules are respectively located in different processors in any combination form.

[0107] The processor contains a kernel, and the kernel retrieves the corresponding program unit from the memory. One or more kernels can be set, and the accuracy of the output of the large language model can be improved by adjusting the kernel parameters.

[0108] The memory may include non - permanent memory in the form of computer - readable media, such as random access memory (RAM) and / or non - volatile memory, such as read - only memory (ROM) or flash RAM, and the memory includes at least one storage chip.

[0109] An embodiment of the present invention provides a computer - readable storage medium, which includes a stored program. When the program runs, it controls the device where the computer - readable storage medium is located to execute the text review method.

[0110] Specifically, the text review method includes:

[0111] Step S201: Obtain the content to be reviewed and determine the initial review result. The content to be reviewed includes at least the text information to be reviewed, and the initial review result represents the result after a preliminary review of the content to be reviewed.

[0112] Step S202: Respectively use the content to be reviewed and the initial review result as keywords to perform searches in the feature model to obtain the first search result and the second search result, and respectively use the content to be reviewed and the initial review result as keywords to perform searches in the search engine to obtain the third search result and the fourth search result. The feature model and the search engine both include historical review content and corresponding historical review results.

[0113] Step S203: Determine the total search result according to the first search result, the second search result, the third search result, and the fourth search result, and output the review result according to the total search result to complete the text review of the content to be reviewed.

[0114] An embodiment of the present invention provides an electronic device, which includes a processor, a memory, and a program stored on the memory and executable on the processor. When the processor executes the program, it implements at least the following steps:

[0115] Step S201: Obtain the content to be reviewed and determine the initial review result. The content to be reviewed includes at least the text information to be reviewed, and the initial review result represents the result after a preliminary review of the content to be reviewed.

[0116] Step S202: Respectively use the content to be reviewed and the initial review result as keywords to perform searches in the feature model to obtain the first search result and the second search result, and respectively use the content to be reviewed and the initial review result as keywords to perform searches in the search engine to obtain the third search result and the fourth search result. The feature model and the search engine both include historical review content and corresponding historical review results.

[0117] Step S203: Determine the total search result based on the first search result, the second search result, the third search result, and the fourth search result, and output a review result based on the total search result to complete the text review of the content to be reviewed.

[0118] The devices in this article can be servers, PCs, PADs, mobile phones, etc.

[0119] This application also provides a computer program product, including a computer program which, when executed by a processor, implements the steps of the methods in the various embodiments of this application:

[0120] Step S201: Obtain the content to be reviewed and determine an initial review result, where the content to be reviewed includes at least the text information to be reviewed, and the initial review result represents the result after a preliminary review of the content to be reviewed;

[0121] Step S202: Respectively use the content to be reviewed and the initial review result as keywords to conduct searches in the feature model to obtain a first search result and a second search result, and respectively use the content to be reviewed and the initial review result as keywords to conduct searches in the search engine to obtain a third search result and a fourth search result, where both the feature model and the search engine include historical review content and corresponding historical review results;

[0122] Step S203: Determine the total search result based on the first search result, the second search result, the third search result, and the fourth search result, and output a review result based on the total search result to complete the text review of the content to be reviewed.

[0123] Obviously, those skilled in the art should understand that the above-mentioned modules or steps of the present invention can be implemented by a general-purpose computing device. They can be concentrated on a single computing device or distributed on a network composed of multiple computing devices. They can be implemented by program codes executable by the computing device, so that they can be stored in a storage device and executed by the computing device. And in some cases, the steps shown or described herein can be executed in a different order, or they can be separately made into individual integrated circuit modules, or multiple modules or steps among them can be made into a single integrated circuit module for implementation. In this way, the present invention is not limited to any specific combination of hardware and software.

[0124] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0125] The present application is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as the combination of flows and / or blocks in the flowchart and / or block diagram, can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for realizing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0126] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device that realizes the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0127] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for realizing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0128] In a typical configuration, a computing device includes one or more processors (CPUs), an input / output interface, a network interface, and a memory.

[0129] The memory may include non-permanent memory in the form of computer-readable media, random access memory (RAM), and / or non-volatile memory such as read-only memory (ROM) or flash memory (flash RAM). The memory is an example of a computer-readable medium.

[0130] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0131] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0132] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

[0133] 1) In the text review method of this application, the content to be reviewed is obtained, and the initial review result is determined. The content to be reviewed and the initial review result are respectively used as keywords to perform searches in the feature model, obtaining the first search result and the second search result. The content to be reviewed and the initial review result are respectively used as keywords to perform searches in the search engine, obtaining the third search result and the fourth search result. The overall search result is determined based on the first search result, the second search result, the third search result, and the fourth search result, and the review result is output according to the overall search result to complete the text review of the content to be reviewed. Compared with the prior art where large models are not applied in the field of text review and the generated content is to some extent uncontrollable with a certain "hallucination" problem, in this application, by separately performing searches in the feature model and the search engine on the content to be reviewed and the initial review result, and then re-outputting the review result based on the above multiple search results, the controllability and accuracy of the finally output review result are ensured. Therefore, it can solve the problem of low accuracy of the results generated by language models in the field of text review in the prior art and achieve the effect of improving the accuracy of language models.

[0134] 2) In the text review device of this application, the content to be reviewed is obtained, and the initial review result is determined. The content to be reviewed and the initial review result are respectively used as keywords to perform searches in the feature model, obtaining the first search result and the second search result. The content to be reviewed and the initial review result are respectively used as keywords to perform searches in the search engine, obtaining the third search result and the fourth search result. The overall search result is determined based on the first search result, the second search result, the third search result, and the fourth search result, and the review result is output according to the overall search result to complete the text review of the content to be reviewed. Compared with the prior art where large models are not applied in the field of text review and the generated content is to some extent uncontrollable with a certain "hallucination" problem, in this application, by separately performing searches in the feature model and the search engine on the content to be reviewed and the initial review result, and then re-outputting the review result based on the above multiple search results, the controllability and accuracy of the finally output review result are ensured. Therefore, it can solve the problem of low accuracy of the results generated by language models in the field of text review in the prior art and achieve the effect of improving the accuracy of language models.

[0135] The above are only the preferred embodiments of this application and are not used to limit this application. For those skilled in the art, this application can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application shall be included within the protection scope of this application.

Claims

1. A text review method, characterized in that, Including: Obtain the content to be reviewed and determine the initial review result. Among them, the content to be reviewed at least includes the text information to be reviewed, and the initial review result represents the result after a preliminary review of the content to be reviewed; Respectively use the content to be reviewed and the initial review result as keywords to perform a search in the feature model to obtain the first search result and the second search result, and respectively use the content to be reviewed and the initial review result as keywords to perform a search in the search engine to obtain the third search result and the fourth search result. Among them, both the feature model and the search engine include historical review content and corresponding historical review results; Determine the total search result according to the first search result, the second search result, the third search result and the fourth search result, and output the review result according to the total search result to complete the text review of the content to be reviewed.

2. The text review method according to claim 1, characterized in that Determining the initial review result includes: Obtain the target language model, where the target language model is trained using multiple groups of data, and each group of data in the multiple groups of data includes the historical review content and the historical review result; Obtain the prompt words, input the prompt words and the content to be reviewed into the target language model, and analyze the content to be reviewed through the target language model to obtain the initial review result.

3. The text review method according to claim 1, characterized in that, Respectively use the content to be reviewed and the initial review result as keywords to perform a search in the feature model to obtain the first search result and the second search result, including: Use the feature model to extract the content features of the historical review content to obtain the first content feature, and use the feature model to extract the result features of the historical review result to obtain the first result feature; Input the content to be reviewed into the feature model to use the feature model to extract the content features of the content to be reviewed to obtain the second content feature; Input the initial review result into the feature model to use the feature model to extract the result features of the initial review result to obtain the second result feature; Calculate the similarity between the first content feature and the second content feature to obtain the first similarity, and arrange the first similarity in a preset order to obtain the first search result corresponding to the content to be reviewed; Calculate the similarity between the first result feature and the second result feature to obtain the second similarity, and arrange the second similarity in the preset order to obtain the second search result corresponding to the initial review result.

4. The text review method according to claim 1, wherein Respectively use the content to be reviewed and the initial review result as keywords to perform a search in the search engine to obtain the third search result and the fourth search result, including: Calculate the text similarity between the historical review content and the content to be reviewed through the inverted index to obtain the third similarity; Calculate the text similarity between the historical review result and the initial review result through the inverted index to obtain the fourth similarity; Arrange the third similarity in a preset order to obtain the third search result corresponding to the content to be reviewed; Arrange the fourth similarity degree in the preset order to obtain the fourth retrieval result corresponding to the content to be reviewed.

5. The text review method according to claim 1, characterized in that, Determine the total retrieval result according to the first retrieval result, the second retrieval result, the third retrieval result and the fourth retrieval result, including: Obtain the weight corresponding to each retrieval result, and obtain the similarity degree corresponding to each retrieval result, where the retrieval results include the first retrieval result, the second retrieval result, the third retrieval result and the fourth retrieval result; Calculate the product of the weight and the similarity degree corresponding to each retrieval result to obtain the total similarity degree corresponding to each retrieval result, and determine the total retrieval result according to the total similarity degree.

6. The text review method according to claim 5, characterized in that, Determine the total retrieval result according to the total similarity degree, including: Merge the first retrieval result, the second retrieval result, the third retrieval result and the fourth retrieval result to obtain a total retrieval item; In the case where there are identical retrieval items in the total retrieval item, take the highest total similarity degree as the total similarity degree of the retrieval item; Sort the total retrieval items in descending order according to the total similarity degree, and determine the first preset threshold number of total retrieval items corresponding to the total similarity degree as the total retrieval result.

7. The text review method according to claim 1, characterized in that Output the review result according to the total retrieval result, including: Obtain a prompt word, and input the prompt word, the content to be reviewed and the total retrieval result into a target language model to use the target language model to review the content to be reviewed again and output the review result.

8. A text review device, characterized in that, Include: A determination unit for obtaining the content to be reviewed and determining an initial review result, where the content to be reviewed includes at least text information to be reviewed, and the initial review result represents the result after a preliminary review of the content to be reviewed; A retrieval unit for respectively using the content to be reviewed and the initial review result as keywords to perform retrieval in a feature model to obtain a first retrieval result and a second retrieval result, and respectively using the content to be reviewed and the initial review result as keywords to perform retrieval in a search engine to obtain a third retrieval result and a fourth retrieval result, where both the feature model and the search engine include historical review content and corresponding historical review results; An output unit for determining a total retrieval result according to the first retrieval result, the second retrieval result, the third retrieval result and the fourth retrieval result, and outputting a review result according to the total retrieval result to complete the text review of the content to be reviewed.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, where when the program runs, it controls the device where the computer-readable storage medium is located to execute the text review method according to any one of claims 1 to 7.

10. An electronic device, characterized in that, Include: One or more processors, a memory, and one or more programs, where the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include those for executing the text review method according to any one of claims 1 to 7.

Images