A retrieval method, system and related devices

Abstract

The application provides a retrieval method, system and related device, the method can include the following steps: obtaining the query text input by the user, then obtaining a plurality of candidate documents with high text relevance to the query text, then determining the legal relevance between each candidate document and the query text, and sorting the plurality of candidate documents to obtain a sorting result, which is displayed to the user, the system first screens out a plurality of candidate documents according to the text relevance, and then sorts the plurality of candidate documents according to the legal relevance, thereby avoiding the problem that the user obtains the same text but non-relevant cases, improving the retrieval accuracy and improving the user experience.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI), and in particular to a retrieval method, system, and related equipment. Background Technology

[0002] Case retrieval refers to the process of searching for related cases from a legal case corpus given a given query case. Since related cases can serve as references influencing judgments and even directly influence the final outcome, case retrieval is crucial for ensuring fairness in the legal field. It not only helps legal professionals provide legal services more efficiently but also enables non-legal professionals to gain a more professional and clear understanding of the legal issues involved in a case.

[0003] However, case retrieval often involves numerous instances where the text is identical but the cases are not related. For example, in cases of traffic accidents and intentional injury, the text regarding "different disability levels" may be identical, but the two cases are not related. Therefore, case retrieval not only needs to determine textual similarity but also needs to identify similarities in legal issues and procedural methods, resulting in poor accuracy and a poor user experience. Summary of the Invention

[0004] This application provides a retrieval method, system, and related equipment to solve the problem of poor accuracy in similar case retrieval and improve the user experience.

[0005] Firstly, a retrieval method is provided, which includes the following steps: obtaining query text input by a user; obtaining multiple candidate documents based on the query text, wherein the textual relevance between the multiple candidate documents and the query text is higher than a threshold; determining the legal relevance between each candidate document and the query text; sorting the multiple candidate documents according to the legal relevance; obtaining the sorting result; and displaying the sorting result to the user.

[0006] The method described in the first aspect involves obtaining the query text input by the user, then obtaining multiple candidate documents with high textual relevance to the query text, determining the legal relevance of each candidate document to the query text, and sorting the multiple candidate documents to obtain a ranking result, which is then displayed to the user. This system first filters multiple candidate documents according to textual relevance, and then sorts the multiple candidate documents according to legal relevance, thereby avoiding the problem of users receiving cases with identical text but not related cases, improving search accuracy, and enhancing the user experience.

[0007] In one possible implementation, when determining the legal relevance between each candidate document and the query text among multiple candidate documents, each candidate document and the query text can be input into a legal relevance model to obtain the legal relevance between each candidate document and the query text. The legal relevance model is obtained by training the AI ​​model using a sample set, which includes input samples and labels of the input samples. The input samples include query samples and candidate document samples, and the labels of the input samples include the legal relevance between the candidate document samples and the query samples.

[0008] The above implementation method uses a legal relevance model to determine the legal relevance between each candidate document and the query text. This allows users to know the legal relevance between the candidate document and their input query text, avoiding the problem of users getting identical texts instead of relevant cases, thus improving search accuracy and user experience.

[0009] In one possible implementation, the labels of the input samples include a first label and a second label. The first label is used to indicate the relevance of the essential facts between the candidate document sample and the query sample, and the second label is used to indicate the relevance of the case facts between the candidate document sample and the query sample.

[0010] The above implementation method uses a sample set containing the first label and the second label to train the model, so that the trained legal relevance model has the ability to predict the relevance of the essential facts and the case facts between the candidate documents and the query text, avoiding the problem that users get the same text but not related cases, improving the retrieval accuracy and improving the user experience.

[0011] In one possible implementation, the legal relevance model includes a feature extraction network and a prediction function. When determining the legal relevance between each candidate document and the query text among multiple candidate documents, each candidate document can first be split into multiple segments. The multiple segments are then input into the feature extraction network to obtain multiple semantic features corresponding to the multiple segments. The multiple semantic features corresponding to the multiple segments are then aggregated to obtain an aggregation result. Finally, the aggregation result is input into the prediction function to obtain the legal relevance between each candidate document and the query text.

[0012] The above implementation method splits the candidate document into multiple segments, extracts the semantic features of each segment, and then aggregates the semantic features of each segment to obtain an aggregated result. The aggregated result is then input into the prediction function to obtain the prediction result. In this way, by processing each candidate document sample in a windowed block, the long judicial document is transformed into an aggregated vector. This avoids the reduction in the accuracy and efficiency of model training due to the excessive length of the judicial document text, and improves the training efficiency and accuracy of the model.

[0013] In one possible implementation, the sample set is obtained after data augmentation of the sample data. The data augmentation scheme includes: when the second sample is a candidate document sample of the first sample, the first sample is a candidate document sample of the second sample. Specifically, the first input sample includes the first sample and the second sample, where the first sample is the query sample and the second sample is a candidate document sample of the first sample. The label of the input sample includes the relevance between the second sample and the first sample. After data augmentation, the newly added second input sample includes the second sample and the first sample, where the second sample is the query sample and the first sample is a candidate document sample of the second sample. The label of the newly added second input sample includes the relevance between the second sample and the first sample; that is, the label of the newly added second input sample inherits the label of the first input sample.

[0014] For example, the first input sample includes query sample A and candidate document sample B, with B labeled (1,1). After data augmentation, the newly added sample is: the second input sample includes query sample B and candidate document sample A, with A labeled (1,1). The above examples are for illustration only and are not intended to be specific.

[0015] In the field of case retrieval, the above implementation method requires professional legal personnel to annotate the samples, which can further reduce the cost of sample annotation. By using data augmentation to obtain more new samples from a small number of samples, the cost of sample annotation can be reduced and the efficiency of sample acquisition can be improved.

[0016] In one possible implementation, the data augmentation scheme further includes: when multiple second samples are candidate document samples of the first sample, the candidate document samples of any one of the multiple second samples for the target sample include the first sample and other second samples besides the target sample. Specifically, the first input sample includes the first sample and multiple second samples, where the first sample is a query sample and the multiple second samples are candidate document samples of the first sample. The label of the first input sample includes the relevance between each second sample and the first sample. After data augmentation, the newly added second input sample includes the target sample, the first sample, and other second samples besides the target sample. The target sample is a query sample, and the first sample and other second samples are candidate document samples of the target sample. The label of the second input sample includes the relevance between other second samples and the first sample, as well as the relevance between the first sample and the target sample. In other words, the label of the newly added second input sample inherits the label of the first input sample. Specifically, the above label can be the highest-level label, i.e., the highly relevant label in Table 1, meaning that both the essential facts and the case facts are relevant.

[0017] In the field of case retrieval, the above implementation method requires professional legal personnel to annotate the samples, which can further reduce the cost of sample annotation. By using data augmentation to obtain more new samples from a small number of samples, the cost of sample annotation can be reduced and the efficiency of sample acquisition can be improved.

[0018] In one possible implementation, the textual relevance between candidate documents and query text is determined based on the probability of words and pairs of words in the query text appearing in candidate documents.

[0019] In practice, when determining the probability of a word / phrase appearing in a document, it can be based on the first frequency of the word / phrase appearing in candidate documents and the second frequency of the word / phrase appearing in all documents in the search set. It should be understood that if a word / phrase appears many times in all documents, it is likely a common term with low distinguishing power between documents. Conversely, if a word / phrase appears only in a few documents, it is likely a specialized term in a specific field with high distinguishing power between documents. Therefore, combining the first and second frequencies to determine the probability of a word / phrase better reflects its importance in the document and the textual relevance between documents, thus improving the accuracy of determining the textual relevance between documents and query text, and ultimately improving the accuracy and efficiency of case retrieval.

[0020] For example, a word 't' may appear infrequently in document 'd' but more frequently in other documents. Its frequency may be low, but its probability may be high because it appears frequently across the entire retrieval set. Therefore, calculating the probability of a word / phrase in a document allows for a more comprehensive consideration of the importance and likelihood of its occurrence, thereby improving the accuracy and effectiveness of the word-based language model.

[0021] Optionally, assuming the query text includes multiple words and multiple pairs of words, after determining the first probability of each word in the candidate documents and the second probability of each pair of words in the candidate documents, a third probability of each word in the query text can be obtained by combining the second probability of each pair of words with the first probability of each word. The textual relevance between the candidate documents and the query text is determined based on the multiple third probabilities of multiple pairs of words appearing in the candidate documents, and multiple candidate documents are obtained based on the textual relevance. For example, the third probability can be obtained using a linear weighted method, based on the first and second probabilities. The textual relevance of each candidate document can be obtained based on the product of the multiple third probabilities of multiple pairs of words. The above examples are for illustrative purposes only and are not intended to limit the scope of the application.

[0022] In the above implementation, this application combines word and word language models in a linear weighted manner to obtain the third probability of two adjacent words appearing simultaneously in the candidate document. Then, the probabilities of two adjacent words appearing simultaneously in the candidate document are multiplied to obtain the text relevance between the candidate document and the query text. If the text relevance of the candidate document is determined by directly multiplying the first probability of a word and the second probability of a word, the number of words and words is very large, which not only results in a very large amount of computation, but also easily leads to the first probability or the second probability being 0, which makes it impossible to guarantee the accuracy of the text relevance. This approach not only avoids excessive computation and the situation of a probability of 0, but also effectively balances the importance of words and words, thereby improving the accuracy and effectiveness of the text relevance calculation.

[0023] In one possible implementation, the textual relevance between candidate documents and query text is determined based on the length of the query text and the frequency of words in the query text appearing in the candidate documents.

[0024] In practice, the textual relevance between candidate documents and query text can be determined using the BM25 algorithm. The user-input query text may include multiple query terms, which can be single words, multiple words, phrases, or sentences. The BM25 algorithm determines the weight of each query term based on its frequency in candidate documents, its frequency in the retrieved text, and the length of the query text. The textual relevance between candidate documents and query text is determined based on the weights of the query terms contained in the candidate documents; the higher the weight of the query terms in a candidate document, the higher the textual relevance between the candidate document and query text. Finally, based on the textual relevance between candidate documents and query text, the aforementioned candidate documents are obtained.

[0025] The above implementation determines the text relevance between candidate documents and query text based on the frequency of query terms appearing in candidate documents and the length of query text. In this way, the more times the query terms appear in candidate documents and the shorter the length of query text, the higher the relevance between candidate documents and query text. Since the length of query text is taken into account, the occurrence of queries with more query terms receiving more weight is avoided. The text relevance determined in this way is highly accurate, and the steps are simple and computationally efficient, making it suitable for large-scale information retrieval.

[0026] In one possible implementation, the text relevance model can be based on the BM25 algorithm and the LMIR algorithm. Specifically, the LMIR algorithm is used to obtain the first text relevance between candidate documents and the query text, and the BM25 algorithm is used to obtain the second text relevance. Multiple candidate documents are then determined based on the linear weighted result of the first and second text relevance. The weights of the first and second text relevance can be adjusted according to the actual application scenario. For example, when the query text is short, the weight of the first text relevance is greater than the weight of the second text relevance; when the query text is long, the weight of the second text relevance is greater than the weight of the first text relevance. It should be understood that the above examples are for illustrative purposes only and are not intended to limit the scope of this application.

[0027] The above implementation method uses the LMIR algorithm to determine the text relevance between candidate documents and query text based on the probability of words and pairs of words appearing in candidate documents, and the BM25 algorithm to determine the text relevance between query text and candidate documents based on the length of query text and the relevance between words in query text and candidate documents. Combining the two algorithms to determine the text relevance between candidate documents and query text not only has high accuracy, but can also handle different types of queries, and has advantages such as globality, adaptability and simplicity.

[0028] Secondly, a retrieval system is provided, comprising: a candidate generation unit for obtaining query text input by a user, obtaining multiple candidate documents based on the query text, wherein the textual relevance between the multiple candidate documents and the query text is higher than a threshold; a sorting unit for determining the legal relevance between each candidate document and the query text, sorting the multiple candidate documents according to legal relevance, and obtaining a sorting result; and a sorting unit for displaying the sorting result to the user.

[0029] The system described in the second aspect obtains the query text input by the user, then obtains multiple candidate documents with high textual relevance to the query text, determines the legal relevance of each candidate document to the query text, and sorts the multiple candidate documents to obtain a ranking result, which is then displayed to the user. This system first filters multiple candidate documents according to textual relevance, and then sorts the multiple candidate documents according to legal relevance, thereby avoiding the problem of users receiving cases with the same text but not related cases, improving search accuracy, and enhancing the user experience.

[0030] In one possible implementation, a sorting unit is used to input each candidate document and query text into a legal relevance model to obtain the legal relevance between each candidate document and the query text. The legal relevance model is obtained by training an AI model using a sample set, which includes input samples and labels for the input samples. The input samples include query samples and candidate document samples, and the labels for the input samples include the legal relevance between the candidate document samples and the query samples.

[0031] In one possible implementation, the labels of the input samples include a first label and a second label. The first label is used to indicate the relevance of the essential facts between the candidate document sample and the query sample, and the second label is used to indicate the relevance of the case facts between the candidate document sample and the query sample.

[0032] In one possible implementation, the relevance model includes a feature extraction network and a prediction function, a ranking unit for splitting each candidate document into multiple segments, a ranking unit for inputting the multiple segments into the feature extraction network to obtain multiple semantic features corresponding to the multiple segments, a ranking unit for aggregating the multiple semantic features corresponding to the multiple segments to obtain an aggregation result, and a ranking unit for inputting the aggregation result into the prediction function to obtain the legal relevance between each candidate document and the query text.

[0033] In one possible implementation, the sample set is obtained by augmenting the sample data. The data augmentation schemes include: when the second sample is a candidate document sample of the first sample, the first sample is a candidate document sample of the second sample.

[0034] In one possible implementation, the scheme used for data augmentation also includes: when multiple second samples are candidate document samples of the first sample, the candidate document samples of any one of the multiple second samples for the target sample include the first sample and other second samples other than the target sample.

[0035] In one possible implementation, the textual relevance between candidate documents and query text is determined based on the probability of words and pairs of words in the query text appearing in candidate documents.

[0036] In one possible implementation, the textual relevance between candidate documents and query text is determined based on the length of the query text and the frequency of words in the query text appearing in the candidate documents.

[0037] Thirdly, a computing device is provided, comprising a memory and a processor, wherein the memory is used to store instructions and the processor is used to execute the instructions to implement the method as described in the first aspect.

[0038] Fourthly, a computing device cluster is provided, the computing device cluster including at least one computing device, each of the at least one computing device including a processor and a memory, the processor of the at least one computing device being configured to execute instructions stored in the memory of the at least one computing device to cause the computing device cluster to implement the method as described in the first aspect.

[0039] Fifthly, a computer-readable storage medium is provided, wherein instructions are stored therein, and the instructions are executed by a computing device or a cluster of computing devices to implement the method described in the first aspect.

[0040] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of the architecture of a retrieval system provided in this application;

[0042] Figure 2 This is a flowchart illustrating the steps of a retrieval method provided in this application;

[0043] Figure 3 This is a flowchart illustrating the steps involved in determining the textual relevance between candidate documents and query text in a retrieval method provided in this application;

[0044] Figure 4 This is a flowchart illustrating the steps involved in training a legal relevance model as provided in this application.

[0045] Figure 5 This is a schematic diagram of the structure of a computing device provided in this application;

[0046] Figure 6 This is a schematic diagram of the structure of a computing device cluster provided in this application;

[0047] Figure 7 This is a schematic diagram of another computing device cluster provided in this application. Detailed Implementation

[0048] The technical solutions of the embodiments of the present invention will now be described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0049] Consistent judgments in similar cases reflect the public's most basic pursuit of judicial fairness and are a fundamental principle that should be upheld in a modern society governed by the rule of law. Different judgments in similar cases can easily lead to public skepticism about the credibility of the judiciary. Therefore, judges, when handling cases, need to conduct case searches for those lacking clear rules of judgment or for which unified rules have not yet been established. Case search refers to retrieving relevant cases from a legal case corpus given a given case. Through case search, judges can achieve consistent judgments in similar cases, uphold judicial authority, and promote the uniformity of legal application; legal professionals can provide more efficient legal services to clients; and non-legal professionals can gain a more professional and clear understanding of the legal issues involved in the case.

[0050] However, case retrieval differs significantly from traditional retrieval methods (such as web page retrieval and text search), presenting new challenges. First, the relevance assessment mechanism in case retrieval differs considerably from traditional retrieval. Case retrieval often involves scenarios where the text is identical but the cases are not related. For example, in traffic accidents and intentional injury cases, the text regarding "different disability levels" may be identical, but the two cases are not related. Second, case retrieval models require manual annotation of the sample set during machine learning. High-quality case samples are scarce, and manual annotation by legal professionals is required, resulting in a high annotation threshold and slow processing speed. Third, legal documents are lengthy, averaging several thousand words, far exceeding the processing capacity of typical AI models.

[0051] In summary, case retrieval is crucial for ensuring fairness in the legal field. However, current case retrieval technologies have many shortcomings, easily retrieving a large number of candidate documents with identical text but unrelated cases, resulting in poor accuracy in case retrieval.

[0052] To address the issue of poor accuracy in similar case retrieval, this application provides a retrieval system. This system can acquire the query text input by the user, then obtain multiple candidate documents with high textual relevance to the query text, determine the legal relevance of each candidate document to the query text, and sort the multiple candidate documents to obtain a ranking result, which is then displayed to the user. This system first filters multiple candidate documents according to textual relevance, and then sorts the multiple candidate documents according to legal relevance, thereby avoiding the problem of users obtaining cases with identical text but not related cases, improving retrieval accuracy, and enhancing the user experience.

[0053] Figure 1 This is a schematic diagram of the architecture of a retrieval system provided in this application, such as... Figure 1As shown, the architecture includes a client 100, a retrieval system 200, and a storage system 300. The client 100, retrieval system 200, and storage system 300 are connected by communication links, which can be wired or wireless; this application does not specify a particular connection. The number of clients 100 and storage systems 300 can be one or more. One retrieval system 200 can also establish communication links with multiple clients 100 or multiple storage systems 300; this application does not specify a particular connection.

[0054] Client 100 can be deployed on user-controlled terminal devices or computing devices. These terminal devices can be personal computers, smartphones, handheld processors, tablets, mobile laptops, augmented reality (AR) devices, virtual reality (VR) devices, all-in-one handheld consoles, wearable devices, in-vehicle devices, smart conferencing devices, smart advertising devices, smart home appliances, etc., without specific limitations. The computing device can be a bare metal server (BMS), virtual machine, container, or edge computing device. Here, BMS refers to a general-purpose physical server, such as an ARM server or an x86 server; a virtual machine refers to a complete computer system implemented using Network Functions Virtualization (NFV) technology, simulated in software, with complete hardware system functionality, running in a completely isolated environment; and a container refers to a group of resource-constrained, isolated processes.

[0055] In a specific implementation, the client 100 can be software or an application running on a terminal device or computing device controlled by the user, such as a personal computer (PC) client, a web client accessed through a browser, an application (APP) client running on a mobile device, or an application programming interface (API). This application does not impose any specific limitations.

[0056] The retrieval system 200 and storage system 300 can be deployed on a computing device, which can be a BMS, a virtual machine, or a container. Here, BMS refers to a general-purpose physical server, such as an ARM server or an x86 server; descriptions of virtual machines and containers can be found above and will not be repeated here.

[0057] Optionally, the retrieval system 200 and the storage system 300 may be deployed on the same computing device or on different computing devices; this application does not impose any specific limitations.

[0058] Optionally, the retrieval system 200 and the storage system 300 can also be deployed on a computing device cluster consisting of multiple computing devices, wherein the retrieval system 200 is deployed on a first computing device cluster and the storage system 300 is deployed on a second computing device cluster, or the retrieval system 200 and the storage system 300 are deployed on the same computing device cluster. This application does not make any specific limitations.

[0059] Optionally, the retrieval system 200 and the client 100 can be deployed on different computing devices; or, the retrieval system 200 can be deployed on a computing device or a cluster of computing devices, and the client 100 can be deployed on a terminal device; or, the retrieval system 200 and the client 100 can be deployed on the same computing device. This application does not make any specific limitations.

[0060] For example, the retrieval system 200 and the storage system 300 can be deployed on the same computing device cluster, and the client 100 can be deployed on a terminal device; or, the retrieval system 200 and the storage system can be deployed on different computing devices or computing device clusters, and the client 100 can be deployed on a terminal device; or, the retrieval system 200 and the client 100 can be deployed on the same computing device, and the storage system 300 can be deployed on other computing devices; or, the retrieval system 200, the client 100, and the storage system 300 can be deployed on the same computing device. It should be understood that the above examples are for illustrative purposes only, and the deployment methods of the retrieval system 200, the storage system 300, and the client 100 can be combined arbitrarily, and this application does not impose any specific limitations.

[0061] Optionally, the client 100 can be integrated into judicial service software as a case retrieval module within the software. This judicial service software may also have other functional modules, such as case process management, case trial progress, and legal aid services, etc., which are not specifically limited in this application. The aforementioned judicial service software can provide industry services to public security, procuratorate, court, and law firms. In specific implementation, users can send query text to the retrieval system 200 through the case retrieval module in the judicial service software, and then receive the query results from the retrieval system 200. These query results may include multiple candidate documents related to the query text. In this application scenario, the retrieval system 200 and storage system 300 can be deployed on computing devices or clusters of computing devices owned by public security, procuratorate, court, or law firms. The judicial service software containing the client 100 can be deployed on computing devices or terminal devices owned by public security, procuratorate, court, or law firms.

[0062] Optionally, client 100 can also be integrated into a public cloud console or API as a sub-service of the judicial cloud service. For example, when a user purchases judicial-related cloud services, they can select the case retrieval service. After successfully purchasing cloud services, the user can obtain the case retrieval function provided in this application through the public cloud console or API. In specific implementation, the user can select the case retrieval service through the console or API, send the query text to be queried to the retrieval system 200 through the console or API, and then receive the query results fed back by the retrieval system 200. In this application scenario, the retrieval system 200 and the storage system 300 can be deployed in the public cloud data center. The console where client 100 is located can be the web interface of the user's computing device or terminal device, or the public cloud API; this application does not make specific limitations.

[0063] Optionally, the client 100 can also be packaged as a standalone case retrieval tool, such as a case retrieval application, microservice, or APP. Users can send query text to the retrieval system 200 through this tool and then receive query results from the retrieval system 200. These results may include multiple candidate documents related to the query text. In this application scenario, the retrieval system 200 and storage system 300 can be deployed on the local computing device where the case retrieval tool resides, or they can be deployed on a remote computing device or cluster of computing devices. This remote computing device or cluster of computing devices may be proprietary to the vendor providing the case retrieval tool.

[0064] It should be understood that the aforementioned concept of "private" can refer to a BMS purchased by the user themselves, or a server purchased by the user through a public cloud; this application does not impose any specific limitations. Furthermore, the client 100, retrieval system 200, and storage system 300 can also take many other forms, depending on the actual application scenario, which will not be elaborated upon here.

[0065] Furthermore, the retrieval system 200 can be further divided into multiple unit modules, for example, such as Figure 1 As shown, the retrieval system 200 may include a preprocessing unit 210, a candidate generation unit 220, a ranking unit 230, and a training unit 240. The retrieval system 200 may also include a storage device 250, which may store a text relevance model and a ranking model. It should be understood that... Figure 1This is one exemplary division method. The retrieval system 200 may also include more units. For example, the training unit may include a text relevance model training unit and a ranking model training unit, and the preprocessing unit 210 may include a word segmentation unit and a data augmentation unit. The retrieval system 200 may also include fewer units. For example, the candidate generation unit 220 and the ranking unit 230 may be merged into one unit. The specific division of unit modules can be based on the actual application scenario, and this application does not limit this.

[0066] The preprocessing unit 210, candidate generation unit 220, sorting unit 230, and training unit 240 within the retrieval system 200 can be implemented in software or hardware. For example, the implementation of the preprocessing unit 210 within the retrieval system 200 will be described below. Similarly, the implementation of the candidate generation unit 220, sorting unit 230, and training unit 240 can refer to the implementation of the preprocessing unit 210.

[0067] As an example of a software functional unit, preprocessing unit 210 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, preprocessing unit 210 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0068] As an example of a hardware functional unit, the preprocessing unit 210 may include at least one computing device, such as a server. Alternatively, the preprocessing unit 210 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0069] The functions of each unit module in the retrieval system 200 are explained below.

[0070] The preprocessing unit 210 in the retrieval system 200 is used to receive the query text sent by the client 100, perform data preprocessing on the query text, and send the preprocessed query text to the candidate generation unit 220. The query text may include one or more words, sentences, and paragraphs, and this application does not make specific limitations.

[0071] In this embodiment, the query text can be a case description of the case to be queried. It should be understood that a complete document contains a large amount of legal provisions, reasoning, and judgment results, making it relatively cumbersome to read and requiring more time and effort. A case description, on the other hand, typically summarizes the core content of the case, making it more concise and clear, facilitating a quick understanding of the case. Furthermore, a case description usually highlights the key points and points of contention in the case, making it more conducive to case retrieval. Therefore, using a case description as the query text can more efficiently find relevant cases and improve retrieval efficiency. In specific implementation, the case description may include, but is not limited to, the cause of the case, the basic information of the parties, the relevant legal provisions, and the analysis and evaluation of the evidence materials. This application does not impose specific limitations on this.

[0072] In one embodiment, data preprocessing of the query text includes data cleaning. Data cleaning may include operations such as removing numbers, punctuation, and stop words from the query text. Here, removing numbers means removing numeric characters from the text, and removing punctuation means removing punctuation marks from the text. Since numbers and punctuation are not important for most natural language processing tasks and will increase the task complexity and interfere with the accuracy of text analysis and processing; removing stop words refers to removing words that frequently appear in the text but are not very helpful for text analysis and processing, such as "de", "shi", "zai", etc. Removing stop words can reduce the noise of the text and improve the efficiency and accuracy of subsequent processing. It should be understood that data cleaning also includes other operations that can improve the efficiency and accuracy of natural language processing, such as stemming, word vectorization, etc., which are not specifically limited in this application.

[0073] In specific implementation, the actual means of implementing data cleaning can be determined according to the application scenario and requirements, which are not specifically limited in this application. For example, removing numbers and punctuation can be implemented using regular expressions. By matching punctuation marks, the punctuation marks in the text are replaced with empty strings, thereby removing all punctuation and numbers. Of course, data cleaning can also be implemented by other means, which are not listed one by one here.

[0074] In one embodiment, data preprocessing of the query text further includes word segmentation processing of the query text. Word segmentation processing refers to splitting a continuous text into multiple individual words, dividing the text into units that are easier to analyze and process for subsequent processing and analysis. In specific implementation, word segmentation processing may include rule-based word segmentation processing methods, statistics-based word segmentation processing methods, and deep learning-based word segmentation processing methods. Among them, rule-based word segmentation processing methods rely on pre-defined dictionaries and grammar rules. For example, the jieba word segmentation tool is used for word segmentation. Among them, jieba word segmentation uses the forward maximum matching algorithm based on the prefix dictionary and the reverse maximum matching algorithm based on the suffix dictionary to implement word segmentation; statistics-based word segmentation processing methods refer to statistical analysis through a large number of corpora to learn the boundary information of vocabulary, such as hidden Markov model (HMM), conditional random field (CRF), etc.; deep learning-based word segmentation processing methods are based on neural network models, such as recurrent neural network (RNN), long short-term memory network (LSTM), etc., to perform word segmentation processing on the text.

[0075] It should be understood that the above examples are for illustrative purposes only. This application does not limit the specific means of word segmentation. In different application scenarios and languages, the methods and standards for word segmentation may vary. The choice can be made according to the actual application scenario and needs. This application does not make any specific limitations.

[0076] It should be noted that the preprocessing unit 210 can receive the query text sent by the client 100 after the model has been trained, during the model inference phase. During the model training phase, the preprocessing unit 210 also receives sample data sent by the storage system 300, performs data augmentation on the sample data to obtain a sample set, and then sends the sample set to the training unit 240 for model training. During this phase, if the sample data received by the preprocessing unit 210 has not undergone data cleaning and word segmentation, the preprocessing unit 210 can perform data cleaning and word segmentation on the sample data; if these operations have already been performed, this step can be omitted.

[0077] In practice, the sample set can be a labeled sample set, and the labels of the sample set can be those labeled by judicial practitioners. Data augmentation refers to expanding the labeled sample set to improve the generalization ability and robustness of the model.

[0078] Optionally, the preprocessing unit 210 can also perform data augmentation on the sample data according to a preset data augmentation scheme to obtain a sample set. The sample set includes input samples and labels for the input samples. The input samples include query samples and one or more candidate document samples related to the query samples. The labels are used to indicate the legal relevance between the candidate document samples and the query samples.

[0079] Optionally, different labels are used for different relevance levels. The label can be a relevance value or a relevance level, depending on the actual application scenario. For example, a highly relevant label is 3, a fairly relevant label is 2, a basically irrelevant label is 1, and a completely irrelevant label is 0.

[0080] Optionally, the label includes a first label and a second label. The first label indicates the relevance of the essential facts between the candidate document sample and the query sample, and the second label indicates the relevance of the case facts between the candidate document sample and the query sample. The essential facts in the legal document refer to important facts related to the case, including the identity of the parties, the time, place, cause, process, and result of the case, which form the basis for the judgment. The case facts refer to the specific circumstances of the case, including the means of committing the crime, the victim's injuries, and the witness testimonies, which determine the nature of the case and the determination of criminal liability.

[0081] In practice, candidate document samples with different relevance to the essential facts have different first labels, and candidate document samples with different relevance to the case facts have different second labels. These can be different relevance values ​​or different relevance levels, depending on the actual application scenario. Table 1 is an example table of labels provided in this application. Candidate document samples with both essential and case facts related are labeled (1,1) or (3); candidate document samples with essential but unrelated facts are labeled (1,0) or (2); candidate document samples with unrelated facts but related facts are labeled (0,1) or (1); and candidate document samples with neither essential nor case facts related are labeled (0,0) or (0). It should be understood that Table 1 is for illustrative purposes only, and this application does not impose specific limitations.

[0082] Table 1: Example of Labels

[0083] Label First tag Second tag Example Highly relevant Relevant facts Case facts (3) / (1,1) Comparative Relevance Relevant facts The facts of the case are irrelevant. (2) / (1,0) Basically irrelevant The essential facts are irrelevant. Case facts (1) / (0,1) Completely unrelated The essential facts are irrelevant. The facts of the case are irrelevant. (0) / (0,0)

[0084] Understandably, by setting a first label and a second label, respectively indicating the relevance between the essential facts and case facts of the candidate document sample and the query text, the model can perform prediction and joint learning based on the essential facts and case facts during training, thereby improving the final prediction effect of the model and avoiding the problem of predicting candidate documents with the same text but not related cases, thus improving the user experience.

[0085] In this embodiment, the data augmentation scheme may include: when the second sample is a candidate document sample of the first sample, the first sample is a candidate document sample of the second sample. Specifically, the first input sample includes a first sample and a second sample, where the first sample is a query sample and the second sample is a candidate document sample of the first sample. The labels of the input samples include the relevance between the second sample and the first sample. After data augmentation, the newly added second input sample includes both the second sample and the first sample, where the second sample is a query sample and the first sample is a candidate document sample of the second sample. The labels of the newly added second input sample include the relevance between the second sample and the first sample; that is, the labels of the newly added second input sample inherit the labels of the first input sample.

[0086] For example, the first input sample includes query sample A and candidate document sample B, with B labeled (1,1). After data augmentation, the newly added sample is: the second input sample includes query sample B and candidate document sample A, with A labeled (1,1). The above examples are for illustration only and are not intended to be specific.

[0087] In practice, the above-mentioned tags can be the highest-level tags, that is, the highly relevant tags in Table 1, which means that the essential facts are relevant and the case facts are also relevant.

[0088] Optionally, the data augmentation scheme may further include: when multiple second samples are candidate document samples of the first sample, the candidate document samples of any one of the multiple second samples for the target sample include the first sample and other second samples besides the target sample. Specifically, the first input sample includes the first sample and multiple second samples, where the first sample is a query sample and the multiple second samples are candidate document samples of the first sample. The label of the first input sample includes the relevance between each second sample and the first sample. After data augmentation, the newly added second input sample includes the target sample, the first sample, and other second samples besides the target sample. The target sample is a query sample, and the first sample and other second samples are candidate document samples of the target sample. The label of the second input sample includes the relevance between other second samples and the first sample, as well as the relevance between the first sample and the target sample. In other words, the label of the newly added second input sample inherits the label of the first input sample. Specifically, the above label can be the highest-level label, i.e., the highly relevant label in Table 1, meaning that the essential facts are relevant and the case facts are also relevant.

[0089] For example, the first input sample includes query sample A and candidate document samples B, C, and D, and the labels of B, C, and D are all (1,1). The second input sample added after data augmentation includes query sample B and candidate document samples A, C, and D, and the labels of A, C, and D are all (1,1). The above example is for illustration only and is not intended to be specific.

[0090] It should be noted that the query sample is a case description, while the candidate document sample is a complete document. The case description summarizes the core content of the case as described in the complete document. Therefore, after data augmentation of the sample set, when the second sample, previously a candidate document sample, becomes a query sample in the newly added input sample, it needs to be converted from a complete document to a case description. Similarly, when the first sample becomes a candidate document for the second sample, it needs to be converted from a case description to a complete document. Taking the above example, input sample A includes a case description of case A and a complete document of case B, where case B is a candidate document of case A. After data augmentation, the newly added input sample B includes a case description of case B and a complete document of case A. It should be understood that the above examples are for illustrative purposes only and this application does not impose specific limitations.

[0091] It should be understood that samples in the field of case retrieval are difficult to label, requiring legal professionals to label them, which is inefficient and costly. This application allows a small number of samples to be augmented to obtain more new samples, thereby reducing the cost of sample labeling and improving the efficiency of sample acquisition.

[0092] In practice, a complete document may include a case description. Therefore, after data augmentation, the case description for a new sample can be obtained from the previous complete document. When a complete document does not include a case description, such as an indictment or answer, it may not contain a case description but only a simple summary of the facts and points of contention, such as a summary. In this case, the summary of the complete document can also be used as a query sample, and this application does not impose any specific limitations.

[0093] It should be noted that data augmentation may also include synonym replacement, replacing certain words in the labeled samples with synonyms to expand the diversity of the sample set; it may also include random insertion or deletion, randomly inserting or deleting some words, phrases or sentences in the labeled samples to increase the size and diversity of the sample set; it may also include sentence recombination, recombining sentences in the labeled samples to increase the diversity and complexity of the sample set. It should be understood that the above examples are for illustration purposes, and other data augmentation operations can also be used to expand the labeled sample set. The specific method can be determined according to the actual application scenario and needs, and this application does not impose specific limitations.

[0094] The candidate generation unit 220 receives the query text sent by the preprocessing unit 210, determines the text relevance between each document and the query text, obtains multiple candidate documents according to the text relevance of each document, and sends the multiple candidate documents to the sorting unit 230. These multiple candidate documents can be multiple candidate documents from a retrieval set, which may include a large number of documents.

[0095] In this embodiment, the candidate generation unit 220 can determine the text relevance between each document and the query text based on a text relevance model. The text relevance model can be a model pre-trained by the training unit 240 and stored in the storage device 250. This text relevance model can take the query text as input and output the text relevance between each document and the query text. The text relevance model will be explained below.

[0096] In one possible implementation, the text relevance model can determine the text relevance between candidate documents and the query text based on the probability of words and pairs of words appearing in candidate documents, thereby obtaining multiple candidate documents. Specifically, the text relevance model can be based on a language model for information retrieval (LMIR). LMIR is a language model based on words and pairs of words. The word language model is used to determine the probability of a single word appearing in a document, and the pair of words language model is used to determine the probability of pairs of words appearing in a document. Combining the word and pair of words language models can determine the probability of words and pairs of words contained in the query text appearing in various documents, thereby determining the text relevance between each document and the query text, and thus identifying multiple candidate documents.

[0097] The aforementioned "double words" refer to linguistic units composed of two adjacent words. It should be understood that in a sentence, adjacent words often have certain grammatical and semantic relationships, such as the relationship between subject and predicate, verb and object, synonyms, antonyms, and modifiers. For example, in the sentence "I like to eat apples," "I like" and "like to eat" are both double words, representing the relationship between subject and predicate, and between verb and object. By analyzing double words, we can better capture the grammatical and semantic relationships between words in a sentence, improving the accuracy of the text relevance between each document and the query text, thereby enhancing the accuracy and efficiency of case retrieval.

[0098] Optionally, when determining the probability of a word / two-word appearing in a document, the word / two-word language model can use the first frequency of the word / two-word appearing in candidate documents and the second frequency of the word / two-word appearing in all documents in the retrieval set. It should be understood that if a word / two-word appears many times in all documents, it is likely a common term with low distinguishing power between documents. Conversely, if a word / two-word appears only in a few documents, it is likely a specialized term in a specific field with high distinguishing power between documents. Therefore, combining the first and second frequencies to determine the probability of a word / two-word can better express the importance of the word / two-word in the document and the textual relevance between documents, thereby improving the accuracy of determining the textual relevance between documents and query texts, and ultimately improving the accuracy and efficiency of case retrieval.

[0099] For example, a word 't' may appear infrequently in document 'd' but more frequently in other documents. Its frequency may be low, but its probability may be high because it appears frequently across the entire retrieval set. Therefore, calculating the probability of a word / phrase in a document allows for a more comprehensive consideration of the importance and likelihood of its occurrence, thereby improving the accuracy and effectiveness of the word-based language model.

[0100] Optionally, assuming the query text includes multiple words and multiple pairs of words, after determining the first probability of each word in the candidate documents and the second probability of each pair of words in the candidate documents, a third probability of each word in the query text can be obtained by combining the second probability of each pair of words with the first probability of each word. The textual relevance between the candidate documents and the query text is determined based on the multiple third probabilities of multiple pairs of words appearing in the candidate documents, and multiple candidate documents are obtained based on the textual relevance. For example, the third probability can be obtained using a linear weighted method, based on the first and second probabilities. The textual relevance of each candidate document can be obtained based on the product of the multiple third probabilities of multiple pairs of words. The above examples are for illustrative purposes only and are not intended to limit the scope of the application.

[0101] It should be understood that directly multiplying the first probability of a word and the second probability of a word pair to determine the text relevance of a candidate document would be computationally intensive due to the sheer number of words and words. Furthermore, it could easily result in the first or second probability being zero, compromising the accuracy of the text relevance calculation. This application combines word and word language models using a linear weighting method to obtain the third probability of two adjacent words appearing simultaneously in a candidate document. Then, the probabilities of two adjacent words appearing simultaneously in a candidate document are multiplied to obtain the text relevance between the candidate document and the query text. This not only avoids excessive computation and the possibility of a zero probability but also effectively balances the importance of words and words, thereby improving the accuracy and effectiveness of the text relevance calculation.

[0102] In another possible implementation, the text relevance model can determine the text relevance between the query text and candidate documents based on the length of the query text and the frequency of words in the query text appearing in candidate documents. Specifically, the text relevance model can be based on the BM25 algorithm. The user-input query text may include multiple query terms, which can be single words, multiple words, phrases, or sentences. The BM25 algorithm determines the weight of each query term based on its frequency in candidate documents, its frequency in the retrieved documents, and the length of the query text. The text relevance between the candidate documents and the query text is determined based on the weights of the query terms contained in the candidate documents; the higher the weight of the query terms in a candidate document, the higher the text relevance between the candidate document and the query text. Finally, based on the text relevance between the candidate documents and the query text, the aforementioned candidate documents are obtained.

[0103] It should be understood that the text relevance between candidate documents and query text is determined based on the frequency of query terms appearing in candidate documents and the length of the query text. In this way, the more times the query terms appear in candidate documents and the shorter the length of the query text, the higher the relevance between the candidate documents and query text. Since the length of the query text is taken into account, the occurrence of queries with more query terms receiving more weight is avoided. The text relevance determined in this way is more accurate, and the process is simple and efficient, making it suitable for large-scale information retrieval.

[0104] In another possible implementation, the text relevance model can be based on the BM25 algorithm and the LMIR algorithm. Specifically, the LMIR algorithm is used to obtain the first text relevance between candidate documents and the query text, and the BM25 algorithm is used to obtain the second text relevance. Multiple candidate documents are then determined based on the linear weighted result of the first and second text relevance. The weights of the first and second text relevance can be adjusted according to the actual application scenario. For example, when the query text is short, the weight of the first text relevance is greater than the weight of the second text relevance; when the query text is long, the weight of the second text relevance is greater than the weight of the first text relevance. It should be understood that the above examples are for illustrative purposes only and are not intended to limit the scope of this application.

[0105] It should be understood that the LMIR algorithm determines the text relevance between candidate documents and query text based on the probability of words and pairs of words appearing in candidate documents, while BM25 determines the text relevance between query text and candidate documents based on the length of query text and the relevance between words in query text and candidate documents. Combining the two algorithms to determine the text relevance between candidate documents and query text not only has high accuracy but can also handle different types of queries, and has advantages such as globality, adaptability, and simplicity.

[0106] It should be noted that the number of candidate documents generated by the candidate generation unit 220 can be determined according to the actual application scenario. After obtaining the text relevance between each candidate document and the query text, the candidate documents can be sorted according to the text relevance. Multiple candidate documents can be obtained according to the sorting result. For example, after sorting from high to low, the top k documents can be taken as candidate documents. This application does not make specific limitations.

[0107] The sorting unit 230 is used to determine the legal relevance between each candidate document and the query text, sort multiple candidate documents according to the legal relevance, obtain the sorting result, and return the sorting result to the client 100. The legal relevance includes the similarity between the cases in the candidate documents and the cases described in the query text in terms of legal issues and legal procedures.

[0108] It should be understood that, as mentioned above, case retrieval scenarios not only need to consider the textual relevance between the document and the query text, but also need to identify the similarity between the legal issues and legal procedures described in the document and the query text, in order to avoid the problem of identical documents that are not related, thereby improving the accuracy of case retrieval and enhancing the user experience.

[0109] In one possible implementation, the sorting unit 230 can obtain the legal relevance between each candidate document and the query text based on a legal relevance model. The legal relevance model can be obtained by the training unit 240 after receiving the sample set sent by the preprocessing unit 210, and then training the AI ​​network using the sample set. The sample set includes input samples and labels for the input samples, where the input samples include query samples and candidate documents for the query samples, and the labels include the relevance between the query samples and the candidate documents. Further, the labels include a first label and a second label, where the first label indicates the relevance of the essential facts between the candidate document samples and the query samples, and the second label indicates the degree of relevance between the case facts between the candidate document samples and the query samples. For details, please refer to the description of the sample set in the aforementioned preprocessing unit 210; further elaboration will not be repeated here.

[0110] In specific implementations, the aforementioned AI networks may include convolutional neural networks (CNN), recurrent neural networks (RNN), support vector machines (SVM), decision trees, deep learning neural networks (DNN), long short-term memory (LSTM) networks, etc., and this application does not impose any specific limitations.

[0111] Optionally, the aforementioned legal relevance model may include a feature extraction network and a prediction function. When training the legal relevance model, candidate document samples can be split into multiple segments, each segment can be input into the feature extraction network to obtain the semantic features corresponding to each segment, and then the semantic features of multiple segments can be aggregated to obtain the aggregation result. The aggregation result can then be input into the prediction function to obtain the prediction result. The model parameters of the prediction function can be adjusted according to the difference between the prediction result and the label until the model converges, thus obtaining the trained legal relevance model.

[0112] It should be understood that splitting candidate documents into multiple segments, extracting semantic features from each segment, and then aggregating the semantic features of each segment to obtain an aggregated result, and inputting the aggregated result into a prediction function to obtain a prediction result, can transform long judicial documents into aggregated vectors by processing each candidate document sample in a windowed block manner. This can avoid reducing the accuracy and efficiency of model training due to the excessive length of judicial documents, and improve the training efficiency and accuracy of the model.

[0113] Optionally, the aforementioned feature extraction network may include, but is not limited to, models such as BERT, XLNet, and ALBERT used to extract semantic features from input vectors, and the prediction function may be a one-layer or multi-layer fully connected network; this application does not impose any specific limitations. When aggregating the semantic features of each segment to obtain the aggregation result, the aggregation method may include max pooling, transformer aggregation, etc.; this application does not impose any specific limitations.

[0114] As mentioned above, the labels of the sample set include a first label and a second label. The first label indicates the relevance of the essential facts between the candidate document sample and the query sample, while the second label indicates the relevance of the case facts between the candidate document sample and the query sample. Therefore, after training the legal relevance model based on the user's input query text and multiple candidate documents related to the query text, it can use the cross-entropy function to predict the essential facts and case facts separately and perform joint learning. This enables the trained legal relevance model to predict the relevance of the essential facts and case facts between the candidate document and the query text, avoiding sending candidate documents with the same text but not related cases to the user, thus improving the user experience.

[0115] In this embodiment, after the sorting unit 230 determines the legal relevance between each candidate document and the query text, it can sort multiple candidate documents according to the legal relevance to obtain a sorting result. The sorting can be based on the magnitude of the legal relevance, and then the candidate documents that meet the threshold can be displayed to the user. Alternatively, the legal relevance of all candidate documents can be displayed to the user in the form of a score, or the sorting result can be displayed to the user. The specific method of display can be selected according to the actual application scenario, and this application does not make specific limitations.

[0116] Optionally, after displaying the legal relevance of candidate documents to the user, new samples can be generated based on the user's feedback. For example, the user can choose the legal relevance of the displayed candidate documents, whether they are related to the essential facts of the case or the facts of the case, thereby obtaining a new sample set to incrementally learn the legal relevance model and continuously improve the model's performance.

[0117] In summary, this application provides a retrieval system that can acquire the query text input by the user, then acquire multiple candidate documents with high textual relevance to the query text, determine the legal relevance of each candidate document to the query text, and sort the multiple candidate documents to obtain a ranking result, which is then displayed to the user. This system first filters multiple candidate documents according to textual relevance, and then sorts the multiple candidate documents according to legal relevance, thereby avoiding the problem of users receiving cases with identical text but not related cases, improving retrieval accuracy, and enhancing the user experience.

[0118] Figure 2 This is a flowchart illustrating the steps of a retrieval method provided in this application, which can be applied to, for example... Figure 1 In the retrieval system 200 shown, such as Figure 2 As shown, the method may include the following steps:

[0119] S210: Obtain the query text input by the user. This step can be performed by... Figure 1 The client 100 in the embodiment is implemented.

[0120] The query text may include one or more words, sentences, or paragraphs, and this application does not impose specific limitations. In practice, the query text may be a description of the case to be queried. It should be understood that complete documents contain a large amount of legal provisions, reasoning, and judgment results, making them relatively cumbersome to read and requiring more time and effort. In contrast, a case description typically summarizes the core content of the case, making it more concise and clear, facilitating a quick understanding of the case's situation. Furthermore, a case description usually highlights the key points and points of contention in the case, which is more conducive to case retrieval. Therefore, using a case description as the query text can more efficiently find relevant cases and improve retrieval efficiency. In practice, the case description may include, but is not limited to, the cause of the case, the basic information of the parties involved, the relevant legal provisions, and the analysis and evaluation of the evidence materials, and this application does not impose specific limitations.

[0121] S220: Retrieve multiple candidate documents based on the query text. This step can be performed by... Figure 1 The preprocessing unit 210 and the candidate generation unit 220 in the embodiment are implemented.

[0122] In this embodiment, the query text can be preprocessed first, and then multiple candidate documents can be obtained. Data preprocessing includes data cleaning, which may include removing numbers, punctuation, stop words, etc., from the query text. For details, please refer to [reference needed]. Figure 1 The relevant descriptions in the embodiments will not be repeated here. It should be understood that data cleaning also includes other operations that can improve the efficiency and accuracy of natural language processing, such as stemming and word vectorization, which are not specifically limited in this application.

[0123] In practice, the actual means of data cleaning can be determined according to the application scenario and requirements. This application does not impose any specific limitations. For example, removing numbers and punctuation can be achieved using regular expressions. By matching punctuation marks, the punctuation marks in the text can be replaced with empty strings, thereby removing all punctuation marks and numbers. Of course, data cleaning can also be achieved through other methods, which will not be illustrated here.

[0124] In one embodiment, data preprocessing of the query text further includes word segmentation. Word segmentation refers to dividing a continuous text into multiple individual words, dividing the text into units that are easier to analyze and process, facilitating subsequent processing and analysis. Specifically, word segmentation can include rule-based, statistical, and deep learning-based methods. Rule-based methods rely on predefined dictionaries and grammatical rules, such as using the jieba word segmentation tool, which employs a forward maximum matching algorithm based on a prefix dictionary and a backward maximum matching algorithm based on a suffix dictionary. Statistical methods involve statistical analysis of a large corpus to learn word boundary information, such as HMM and CRF. Deep learning-based methods use neural network models, such as RNN and LSTM, to segment the text.

[0125] It should be understood that the above examples are for illustrative purposes only. This application does not limit the specific means of word segmentation. In different application scenarios and languages, the methods and standards for word segmentation may vary. The choice can be made according to the actual application scenario and needs. This application does not make any specific limitations.

[0126] In one embodiment, after preprocessing the query text, the text relevance between each document and the query text can be determined based on the preprocessed query text. Multiple candidate documents are then obtained according to the text relevance of each document. These multiple candidate documents can be multiple candidate documents in a retrieval set, which may include a large number of documents. Specifically, the text relevance between each document and the query text can be determined based on a text relevance model.

[0127] In one possible implementation, the text relevance model can determine the text relevance between candidate documents and the query text based on the probability of words and pairs of words appearing in candidate documents, thereby obtaining multiple candidate documents. Specifically, the text relevance model can be based on LMIR (Lesson-Like Language Indicators), which is a language model based on words and pairs of words. The word language model determines the probability of a single word appearing in a document, and the pair of words language model determines the probability of pairs of words appearing in a document. Combining the word and pair of words language models can determine the probability of words and pairs of words contained in the query text appearing in each document, thus determining the text relevance between each document and the query text, and ultimately identifying multiple candidate documents.

[0128] In another possible implementation, the text relevance model can determine the text relevance between the query text and candidate documents based on the length of the query text and the frequency of words in the query text appearing in candidate documents. Specifically, the text relevance model can be based on the BM25 algorithm. The user-input query text may include multiple query terms, which can be single words, multiple words, phrases, or sentences. The BM25 algorithm determines the weight of each query term based on its frequency in candidate documents, its frequency in the retrieved documents, and the length of the query text. The text relevance between the candidate documents and the query text is determined based on the weights of the query terms contained in the candidate documents; the higher the weight of the query terms in a candidate document, the higher the text relevance between the candidate document and the query text. Finally, based on the text relevance between the candidate documents and the query text, the aforementioned candidate documents are obtained.

[0129] In another possible implementation, the text relevance model can be based on the BM25 algorithm and the LMIR algorithm. Specifically, the LMIR algorithm is used to obtain the first text relevance between candidate documents and the query text, and the BM25 algorithm is used to obtain the second text relevance. Multiple candidate documents are then determined based on the linear weighted result of the first and second text relevance. The weights of the first and second text relevance can be adjusted according to the actual application scenario. For example, when the query text is short, the weight of the first text relevance is greater than the weight of the second text relevance; when the query text is long, the weight of the second text relevance is greater than the weight of the first text relevance. It should be understood that the above examples are for illustrative purposes only and are not intended to limit the scope of this application.

[0130] It should be understood that the LMIR algorithm determines the text relevance between candidate documents and query text based on the probability of words and pairs of words appearing in candidate documents, while BM25 determines the text relevance between query text and candidate documents based on the length of query text and the relevance between words in query text and candidate documents. Combining the two algorithms to determine the text relevance between candidate documents and query text not only has high accuracy but can also handle different types of queries, and has advantages such as globality, adaptability, and simplicity.

[0131] It should be noted that the number of candidate documents can be determined according to the actual application scenario. After obtaining the text relevance between each candidate document and the query text, the candidate documents can be sorted according to the text relevance. Multiple candidate documents can be obtained according to the sorting results. For example, after sorting from high to low, the top k documents can be taken as candidate documents. This application does not make specific limitations.

[0132] For example, Figure 3This is a flowchart illustrating the steps involved in determining the textual relevance between candidate documents and query text in a retrieval method provided in this application, as shown below. Figure 3 As shown, the steps can be as follows:

[0133] S221: Determine the first frequency of a word in the candidate documents, the second frequency of a word in the search set, the first frequency of a word in the candidate documents, and the second frequency of a word in the search set.

[0134] S222: Determine the first probability of a word appearing in candidate documents based on the first and second frequencies of the word, and determine the second probability of a word appearing in candidate documents based on the first and second frequencies of the word.

[0135] For example, the language model for words and diwords can be represented by the following formula (1):

[0136] P(t|d)=λP mle (t|M d )+(1-λ)P mle (t|M c (1) Where t represents a word / two-word, P mle (t|M d P represents the frequency of t in document d. If t does not appear in document d, P is zero. mle (t|M d ) = 0, P mle (t|M c ) represents the frequency of t in the text of all documents, and 0≤λ≤1 is the linear smoothing factor.

[0137] It should be understood that if a word / phrase appears frequently in all documents, it is likely a common term with low distinguishing power between documents. Conversely, if a word / phrase appears only in a few documents, it is likely a specialized term in a specific field with high distinguishing power. Therefore, combining the first and second frequencies to determine the probability of a word / phrase can better express the importance of the word / phrase in the documents and the textual relevance between documents, thereby improving the accuracy of determining the textual relevance between documents and query texts, and ultimately improving the accuracy and efficiency of case retrieval.

[0138] S223: Combine the second probability of each word with the first probability of each word to obtain the third probability of each word in the query text. For example, the third probability can be obtained by combining the word and word language models in a linear weighted manner, based on the first and second probabilities. That is, the following formula (2):

[0139] P(t i-1 ti |d)=μP1(t i |d)+(1-μ)P2(t i-1 t i |d) (2) where 0≤μ≤1 is the weighting factor, and P1 and P2 are the word and word language models obtained from the previous formula, respectively. Under a certain query, the score of the document is represented by the probability of each document generating the query, and is calculated according to the following formula, where q represents the query, d j Let t represent the j-th document. i-1 t i P(t) represents the two words that appear in query q. i-1 t i |d j The result is obtained through the formula above. It's important to note here that, to avoid invalid values, if t... i-1 t i If the word has never appeared in any document, then ignore it.

[0140] S224: Determine the first text relevance between the candidate document and the query text based on the multiple third probabilities of multiple words appearing in the candidate document.

[0141] For example, the score can be calculated according to the following formula (3). Lm (q,d j This refers to the document d given a query q based on a language model that uses words and pairs of words. j Score:

[0142]

[0143] S225: Determine the second text relevance between candidate documents and query text based on the BM25 algorithm.

[0144] For example, the score calculated by formula (4) below BM25 (q,d j This refers to the document d under a given query q based on the BM25 algorithm. j Score:

[0145]

[0146] Where n represents the length of query q, q i This indicates a query for the i-th word in q. They represent q respectively i In document d j and the frequency of occurrence in query q, For inverse document frequency, if q i If the word has not appeared in any documents, skip it.j | indicates document d j The length of avgdl represents the average length of all documents.

[0147] S226: Determine the text relevance between the candidate document and the query text based on the first and second text relevance scores. In practice, this can be achieved by combining the two methods and applying a linear weighting, as shown in the following formula:

[0148] score final (q,d j )=α·score LMLR (q,d j )+(1-α)·score BM25 (q,d j (5)

[0149] Where 0 ≤ α ≤ 1, represents the weight parameter. The weight parameter can be adjusted according to the actual application scenario; for example, when the query text length is short, the score... LMLR (q,d j The weight α of the score is greater than the score. BM25 (q,d j The weight (1-α) of the query text is used to determine the score when the query text is long. BM25 (q,d j The weight (1-α) of the score is greater than the score. LMLR (q,d j The weight α of ).

[0150] It should be understood that S221 to S226 above are examples of step S220. This application does not limit the above exemplary formula. The above formula can be adapted to different application scenarios and business needs. Examples are not given here.

[0151] S230: Determine the legal relevance between each of the multiple candidate documents and the query text. This step can be performed by... Figure 1 The sorting unit 230 in this embodiment is implemented. The legal relevance includes the similarity between the cases in the candidate documents and the cases described in the query text in terms of legal issues and legal procedural methods.

[0152] It should be understood that, as mentioned above, case retrieval scenarios not only need to consider the textual relevance between the document and the query text, but also need to identify the similarity between the legal issues and legal procedures described in the document and the query text, in order to avoid the problem of identical documents that are not related, thereby improving the accuracy of case retrieval and enhancing the user experience.

[0153] In one possible implementation, the legal relevance between each candidate document and the query text can be obtained based on a legal relevance model. This legal relevance model can be obtained by training the AI ​​network using a sample set, which includes input samples and their labels. The input samples include query samples and their candidate documents, and the labels indicate the relevance between the query samples and the candidate documents.

[0154] Furthermore, the label includes a first label and a second label, wherein the first label is used to indicate the relevance of the essential facts between the candidate document sample and the query sample, and the second label is used to indicate the degree of relevance between the case facts between the candidate document sample and the query sample. For details, please refer to the description of the first label and the second label in the aforementioned preprocessing unit 210, as well as the description of the embodiment in Table 1, which will not be repeated here.

[0155] Optionally, the aforementioned legal relevance model may include a feature extraction network and a prediction function. When training the legal relevance model, candidate document samples can be split into multiple segments, each segment can be input into the feature extraction network to obtain the semantic features corresponding to each segment, and then the semantic features of multiple segments can be aggregated to obtain the aggregation result. The aggregation result can then be input into the prediction function to obtain the prediction result. The model parameters of the prediction function can be adjusted according to the difference between the prediction result and the label until the model converges, thus obtaining the trained legal relevance model.

[0156] It should be understood that splitting candidate documents into multiple segments, extracting semantic features from each segment, and then aggregating the semantic features of each segment to obtain an aggregated result, and inputting the aggregated result into a prediction function to obtain a prediction result, can transform long judicial documents into aggregated vectors by processing each candidate document sample in a windowed block manner. This can avoid reducing the accuracy and efficiency of model training due to the excessive length of judicial documents, and improve the training efficiency and accuracy of the model.

[0157] Optionally, the aforementioned feature extraction network may include, but is not limited to, models such as BERT, XLNet, and ALBERT used to extract semantic features from input vectors, and the prediction function may be a one-layer or multi-layer fully connected network; this application does not impose any specific limitations. When aggregating the semantic features of each segment to obtain the aggregation result, the aggregation method may include max pooling, transformer aggregation, etc.; this application does not impose any specific limitations.

[0158] As mentioned above, the labels of the sample set include a first label and a second label. The first label indicates the relevance of the essential facts between the candidate document sample and the query sample, while the second label indicates the relevance of the case facts between the candidate document sample and the query sample. Therefore, after training the legal relevance model based on the user's input query text and multiple candidate documents related to the query text, it can use the cross-entropy function to predict the essential facts and case facts separately and perform joint learning. This enables the trained legal relevance model to predict the relevance of the essential facts and case facts between the candidate document and the query text, avoiding sending candidate documents with the same text but not related cases to the user, thus improving the user experience.

[0159] For example, Figure 4 This is a flowchart illustrating the steps involved in training a legal relevance model provided in this application, such as... Figure 4 As shown, firstly, the candidate document sample D is divided into multiple segments according to a certain length, as shown in the following formula (6):

[0160] D = {P1, ..., P} n} (6)

[0161] Among them, P i The term refers to a segment of the judicial document D, such as a paragraph or a page; this application does not specify a particular segment.

[0162] Secondly, pre-trained language models such as BERT are used to capture query samples q and P. i Semantic features at the paragraph level The following formula (7) is expressed:

[0163]

[0164] Next, aggregate semantic features at the paragraph level. The semantic relevance representation D of the query sample and the candidate document sample is obtained. cls Aggregation methods include, but are not limited to, max pooling or Transformer aggregation.

[0165] For example, the formula for maximum aggregation can be as follows:

[0166]

[0167] in, express The i-th term in the vector.

[0168] The formula for Transformer aggregation is as follows:

[0169]

[0170] Here, LayerNorm represents regularization of layer parameters, and MultiHead represents a multi-head self-supervised attention module.

[0171] Then, the aggregated feature D obtained after aggregation cls Input the prediction function, obtain the prediction results, and calculate the relevance score between the query sample q and the candidate document sample D as follows:

[0172] r(q,D)=WD cls (10)

[0173] Where W represents a single-layer neural network. It should be understood that the above formula is for illustrative purposes only, and this application does not limit the formula. The formula can be adapted to different application scenarios and business needs, and specific examples are not provided here.

[0174] Finally, the loss function is determined based on the prediction results and labels, and the parameters of the legal relevance model are adjusted according to the loss function until the model converges.

[0175] In practice, the cross-entropy loss function can be used to predict the essential facts and the case facts separately, and then jointly learn them. This enables the trained legal relevance model to predict the essential facts relevance and the case facts relevance between the candidate document and the query text, thus avoiding sending candidate documents with the same text but not related cases to the user and improving the user experience.

[0176] S240: Rank multiple candidate documents according to their legal relevance to obtain a ranking result. This step can be performed by... Figure 1 The sorting unit 230 in the embodiment is implemented.

[0177] In this application embodiment, after determining the legal relevance between each candidate document and the query text, multiple candidate documents can be sorted according to the legal relevance to obtain a sorting result. The sorting can be based on the magnitude of the legal relevance, and then the candidate documents that meet the threshold can be displayed to the user. Alternatively, the legal relevance of all candidate documents can be displayed to the user in the form of a score, or the sorting result can be displayed to the user. The specific method of display can be selected according to the actual application scenario, and this application does not make specific limitations.

[0178] S250: Display the sorting results to the user. This step can be performed by... Figure 1 The sorting unit 230 in the embodiment is implemented.

[0179] Optionally, after displaying the legal relevance of candidate documents to the user, new samples can be generated based on the user's feedback. For example, the user can choose the legal relevance of the displayed candidate documents, whether they are related to the essential facts of the case or the facts of the case, thereby obtaining a new sample set to incrementally learn the legal relevance model and continuously improve the model's performance.

[0180] In summary, this application provides a retrieval system that can acquire the query text input by the user, then acquire multiple candidate documents with high textual relevance to the query text, determine the legal relevance of each candidate document to the query text, and sort the multiple candidate documents to obtain a ranking result, which is then displayed to the user. This system first filters multiple candidate documents according to textual relevance, and then sorts the multiple candidate documents according to legal relevance, thereby avoiding the problem of users receiving cases with identical text but not related cases, improving retrieval accuracy, and enhancing the user experience.

[0181] Figure 5 This is a schematic diagram of the structure of a computing device provided in this application. The computing device 500 can be the retrieval system mentioned above.

[0182] Furthermore, the computing device 500 includes a processor 501, a storage unit 502, a storage medium 503, and a communication interface 504. The processor 501, the storage unit 502, the storage medium 503, and the communication interface 504 communicate via a bus 505, or via other means such as wireless transmission.

[0183] Processor 501 may include any one or more processors such as a central processing unit (CPU), a microprocessor (MP), or a digital signal processor (DSP).

[0184] Examples include a CPU, an NPU, or a combination of a CPU and hardware chips. The aforementioned hardware chips are Application-Specific Integrated Circuits (ASICs), Programmable Logic Devices (PLDs), or combinations thereof. The aforementioned PLDs are Complex Programmable Logic Devices (CPLDs), Field-Programmable Gate Arrays (FPGAs), Generic Array Logic (GALs), or any combination thereof. The processor 501 executes various types of digital storage instructions, such as software or firmware programs stored in the storage unit 502, enabling the computing device 500 to provide a wide range of services.

[0185] In a specific implementation, as one embodiment, the processor 501 includes one or more CPUs, for example... Figure 5 CPU0 and CPU1 are shown in the diagram.

[0186] In a specific implementation, as one example, the computing device 500 also includes multiple processors, for example... Figure 5 The processors 501 and 506 are shown. Each of these processors can be a single-core processor or a multi-core processor. Here, a processor refers to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).

[0187] Storage unit 502 is used to store program code, and its execution is controlled by processor 501 to perform the above-mentioned tasks. Figures 1-6 The processing steps of the retrieval system in any embodiment. The program code includes one or more software units. The one or more software units mentioned above are... Figure 1 The embodiment includes a preprocessing unit, a candidate generation unit, a sorting unit, and a training unit, wherein the preprocessing unit is used to perform... Figure 2 Step S210 and its optional steps in the embodiment are executed by the candidate generation unit. Figure 2 Step S220 in the embodiment and Figure 3 In the embodiments, steps S221 to S226 and their optional steps are executed by the sorting unit. Figure 2 In the embodiments, steps S230 to S250 and their optional steps, the training unit is used to pre-train the text relevance model and legal relevance model required to obtain the candidate generation unit and the ranking unit, which will not be described in detail here.

[0188] Storage unit 502 includes read-only memory and random access memory, and provides instructions and data to processor 501. Storage unit 502 also includes non-volatile random access memory. Storage unit 502 is volatile memory or non-volatile memory, or includes both. The non-volatile memory is read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory is random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are used, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DRRAM). Other examples include hard disks, USB flash drives, flash memory, SD cards, Memory Sticks, etc., where hard disks include hard disk drives (HDDs), solid-state drives (SSDs), and mechanical hard disks (HDDs), etc., which are not specifically limited in this application.

[0189] Storage medium 503 is a carrier for storing data, such as hard disk, USB flash drive, flash memory, SD card, memory stick, etc. The hard disk can be a hard disk drive (HDD), solid state disk (SSD), mechanical hard disk (HDD), etc. This application does not make specific limitations.

[0190] The communication interface 504 can be used to provide information input or output to the processor 501, or alternatively, the communication interface 504 can be used to receive data sent from the outside and / or send data to the outside, and can be a wired interface including such as an Ethernet cable, or a wireless interface (such as Wi-Fi, Bluetooth, general wireless transmission, etc.), or alternatively, the communication interface 504 may also include a transmitter (such as an RF transmitter, antenna, etc.) or a receiver coupled to the interface.

[0191] Bus 505 is a Peripheral Component Interconnect Express (PCIe) bus, or an Extended Industry Standard Architecture (EISA) bus, Unified Bus (Ubus or UB), Compute Express Link (CXL), Cache Coherent Interconnect for Accelerators (CCIX), etc. Bus 505 is divided into address bus, data bus, and control bus. In addition to the data bus, Bus 505 also includes a power bus, control bus, and status signal bus. However, for clarity, all buses are labeled as Bus 505 in the diagram.

[0192] It needs to be explained that, Figure 5 This is merely one possible implementation of an embodiment of this application. In practical applications, the computing device 500 may include more or fewer components, and this is not a limitation. For content not shown or described in the embodiments of this application, please refer to the foregoing. Figures 1-4 The relevant descriptions in the embodiments will not be repeated here.

[0193] Figure 6This is a schematic diagram of a computing device cluster provided in this application, which includes one or more computing devices 600. Figure 6 As shown, the memory 603 in each computing device 600 may store the same instructions required for executing the retrieval system. These instructions can implement one or more software units. These one or more software units are... Figure 1 The embodiment includes a preprocessing unit, a candidate generation unit, a sorting unit, and a training unit, wherein the preprocessing unit is used to perform... Figure 2 Step S210 and its optional steps in the embodiment are executed by the candidate generation unit. Figure 2 Step S220 in the embodiment and Figure 3 In the embodiments, steps S221 to S226 and their optional steps are executed by the sorting unit. Figure 2 In the embodiments, steps S230 to S250 and their optional steps, the training unit is used to pre-train the text relevance model and legal relevance model required to obtain the candidate generation unit and the ranking unit, which will not be described in detail here.

[0194] The computing device 600 includes a processor 601, a communication interface 602, a memory 603, and a bus 604. Further descriptions of the processor 601, communication interface 602, memory 603, and bus 604 can be found in [reference needed]. Figure 5 The descriptions of processor 501, storage unit 502, storage medium 503, communication interface 504, and bus 505 in the embodiments will not be repeated here.

[0195] like Figure 7 As shown, Figure 7 This is another schematic diagram of the computing device cluster provided in this application. Figure 7 In the implementation shown, the memory 703 in different computing devices 700 can also store different instructions, which are used to execute some functions of the retrieval system respectively. That is, the instructions stored in the memory 703 in different computing devices 700 can be combined to implement the preprocessing unit, candidate generation unit, sorting unit, and training unit, and one or more computing devices 700 can establish a communication connection through an external network or an internal network.

[0196] like Figure 7 As shown, the two computing devices 700A and 700B are connected via a network. Specifically, they are connected to the network through the communication interfaces in each computing device. In this possible implementation, the memory 703 in computing device 700A stores instructions for the functions of the training unit. Meanwhile, the memory 703 in computing device 700B stores instructions for executing the functions of the preprocessing unit, candidate generation unit, and sorting unit. The functional descriptions of each unit can be found in [reference needed]. Figure 6 Examples are not repeated here.

[0197] It needs to be explained that, Figure 7 The implementation shown may be an implementation method when the processing power of the computing device 700B is insufficient, or when the storage space of the computing device 700B is insufficient, or an implementation method under other business scenarios. This application does not make specific limitations.

[0198] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product runs on a computing device or a cluster of computing devices, it causes the computing device or cluster of computing devices to execute a retrieval method.

[0199] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct a computing device to perform a retrieval method, or instruct a cluster of computing devices to perform a retrieval method.

[0200] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented in software, the above embodiments can be implemented, in whole or in part, as a computer program product. A computer program product includes a plurality of computer instructions. When the computer program instructions are loaded or executed on a computer, all or part of the flow or function according to the embodiments of the present invention is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0201] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent repairs or substitutions within the technical scope disclosed in the present invention, and these repairs or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A retrieval method, characterized in that, The method includes: Obtain the query text input by the user, and obtain multiple candidate documents based on the query text, wherein the text relevance between the multiple candidate documents and the query text is higher than a threshold; Each candidate document and query text are input into a legal relevance model to obtain the legal relevance between each candidate document and the query text. The legal relevance model is obtained by training an artificial intelligence (AI) model using a sample set. The sample set includes input samples and labels for the input samples. The input samples include query samples and candidate document samples. The labels for the input samples include a first label and a second label. The first label is used to indicate the relevance of the essential facts between the candidate document sample and the query sample. The second label is used to indicate the relevance between the case facts between the candidate document sample and the query sample. The candidate documents are sorted according to their legal relevance to obtain a sorting result; The sorting results are displayed to the user.

2. The method according to claim 1, characterized in that, The legal relevance model includes a feature extraction network and a prediction function. The step of inputting each candidate document and the query text into the legal relevance model to obtain the legal relevance between each candidate document and the query text includes: Each candidate document is split into multiple fragments; The multiple segments are input into the feature extraction network to obtain multiple semantic features corresponding to the multiple segments; The multiple semantic features corresponding to the multiple segments are aggregated to obtain the aggregation result; The aggregation result is input into the prediction function to obtain the legal relevance between each candidate document and the query text.

3. The method according to claim 2, characterized in that, The sample set is obtained by augmenting the sample data. The data augmentation scheme includes: when the second sample is a candidate document sample of the first sample, the first sample is a candidate document sample of the second sample.

4. The method according to claim 3, characterized in that, The data augmentation scheme further includes: when multiple second samples are candidate document samples of the first sample, the candidate document sample of any one of the multiple second samples is the first sample and other second samples other than the target sample.

5. The method according to claim 4, characterized in that, The text relevance between the candidate documents and the query text is determined based on the probability of words and double words in the query text appearing in the candidate documents.

6. The method according to any one of claims 1 to 5, characterized in that, The text relevance between the candidate documents and the query text is determined based on the length of the query text and the frequency of words in the query text appearing in the candidate documents.

7. A retrieval system, characterized in that, The system includes: A candidate generation unit is used to obtain the query text input by the user and obtain multiple candidate documents based on the query text, wherein the text relevance between the multiple candidate documents and the query text is higher than a threshold. A sorting unit is used to input each candidate document and query text into a legal relevance model to obtain the legal relevance between each candidate document and the query text. The legal relevance model is obtained by training an artificial intelligence (AI) model using a sample set. The sample set includes input samples and labels of the input samples. The input samples include query samples and candidate document samples. The labels of the input samples include a first label and a second label. The first label is used to indicate the relevance of the essential facts between the candidate document samples and the query samples. The second label is used to indicate the relevance between the case facts between the candidate document samples and the query samples. The sorting unit is used to sort the multiple candidate documents according to the legal relevance to obtain a sorting result; The sorting unit is also used to display the sorting results to the user.

8. The system according to claim 7, characterized in that, The legal relevance model includes a feature extraction network and a prediction function; The sorting unit is used to split each candidate document into multiple fragments; The sorting unit is used to input the multiple segments into the feature extraction network to obtain multiple semantic features corresponding to the multiple segments; The sorting unit is used to aggregate multiple semantic features corresponding to the multiple segments to obtain an aggregation result; The sorting unit is used to input the aggregation result into the prediction function to obtain the legal relevance between each candidate document and the query text.

9. The system according to claim 8, characterized in that, The sample set is obtained by augmenting the sample data. The data augmentation scheme includes: when the second sample is a candidate document sample of the first sample, the first sample is a candidate document sample of the second sample.

10. The system according to claim 9, characterized in that, The data augmentation scheme further includes: when multiple second samples are candidate document samples of the first sample, the candidate document sample of any one of the multiple second samples is the first sample and other second samples other than the target sample.

11. The system according to claim 10, characterized in that, The text relevance between the candidate documents and the query text is determined based on the probability of words and double words in the query text appearing in the candidate documents.

12. The system according to any one of claims 7 to 11, characterized in that, The text relevance between the candidate documents and the query text is determined based on the length of the query text and the frequency of words in the query text appearing in the candidate documents.

13. A computing device, characterized in that, The computing device includes a memory and a processor, the memory being used to store instructions, and the processor being used to execute the instructions to implement the method as claimed in any one of claims 1 to 6.

14. A computing device cluster, characterized in that, The computing device cluster includes at least one computing device, each of the at least one computing device including a processor and a memory, wherein the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the computing device cluster to implement the method as claimed in any one of claims 1 to 6.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed by a computing device or a cluster of computing devices, implement the method as claimed in any one of claims 1 to 6.

Images