Rag multi-granularity recall method and system based on large model semantic enhancement

Abstract

This invention relates to the field of data sharing technology, specifically disclosing a multi-granularity recall method and system based on large model semantic enhancement for RAG (Related Aspect-Oriented Generic Data). The method includes receiving a text task set; processing the text in the text task set using a processor with a built-in large model, and synchronously recording the processing progress; comparing the processing progress and processed text between any two processors with built-in large models to construct a private shared database for the two processors; comparing all private shared databases to determine a global shared database; and constructing a matching processing scheme based on the global shared database as a pre-processing process for each processor with a built-in large model. Although the accuracy is slightly insufficient when facing similar tasks, the recall range obtained by the invention is extremely small, greatly improving the response speed with only a slight reduction in accuracy.

Description

Technical Field

[0001] This invention relates to the field of data sharing technology, specifically a RAG multi-granularity recall method and system based on large model semantic enhancement. Background Technology

[0002] Retrieval Augmented Generation (RAG) is a technique that, simply put, retrieves relevant knowledge through retrieval, defines a scope, and then allows a large model to refer to that knowledge to provide a reasonable answer. The core of RAG can be understood as "retrieval + generation," where the process of retrieving and defining the scope is the recall phase.

[0003] Existing RAG technologies mostly employ conventional keyword matching techniques in their recall phase, which is also a standard retrieval function. This approach is certainly feasible and highly efficient when facing new tasks. However, there is another scenario in existing technologies where the text tasks to be processed have a high degree of similarity. For example, users may have very similar needs at a certain stage and in a certain situation. In other words, different users may send similar questions. In this case, the data between different users cannot be shared in the existing recall architecture. This means that there is still room for recall optimization in this situation. Therefore, how to provide an information range determination scheme to improve the interaction speed in repetitive interaction scenarios is the technical problem that this invention aims to solve. Summary of the Invention

[0004] The purpose of this invention is to provide a multi-granularity recall method and system for RAG based on large model semantic enhancement, so as to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: A multi-granularity recall method for RAG based on large model semantic enhancement, the method comprising: It receives a set of text tasks, processes the text in the set based on a processor with a built-in large model, and records the processing progress synchronously. For any two processors with built-in large models, compare the processing progress and the text already processed to construct a private shared database for the two processors with built-in large models; the shared database includes text items and processing progress items; By comparing all private domain shared databases, the global shared database is determined; the total number of data items in the global shared database is a dynamic value. A matching-based processing scheme based on a globally shared database is constructed as a pre-processing step for each built-in large model processor; The processing speed and accuracy of the text task set are periodically obtained, and the total number of data items in the global shared database is updated based on the processing speed and accuracy.

[0006] Furthermore, the step of comparing the processing progress and processed text between any two processors with built-in large models, and constructing a private shared database for the two processors with built-in large models, includes: For any two processors with built-in large models, query the working logs of the processors with built-in large models; The processing time for each text file is queried from the work log. The processing time is then standardized to obtain the standard processing time, which is expressed as a percentage. The texts are initially clustered based on standard time consumption. Each cluster of texts is then compared pairwise, and a second clustering is performed based on the comparison results. For each class of text after secondary clustering, select feature text, read the processing progress of feature text, insert it into the database, insert the name labels of the processors of the two built-in large models into the database, and obtain the private domain shared database of the processors of the two built-in large models.

[0007] Furthermore, during the initial clustering of texts based on standard time consumption, the clustering distance is calculated using the absolute difference in standard time consumption. For each text in the initial clustering, pairwise comparisons are performed. During the secondary clustering process based on the comparison results, the clustering distance is inversely proportional to the text similarity. The clustering condition is that the maximum value of the pairwise distance between any two texts in the clustering result is less than a preset distance threshold. The process of selecting feature text is as follows: Read each text class after the second clustering in sequence. Read the text in that class in sequence, calculate the mean distance of the text to all other texts, and select the text with the smallest mean distance as the feature text.

[0008] Furthermore, the step of comparing all private domain shared databases to determine the global domain shared database includes: Read the N private shared databases that have been created, compare the text items in the N private shared databases, calculate the intersection, and use N as the label of the intersection; Calculate the difference between each private shared database and the intersection. Enumerate and select N-1 difference sets from all the difference sets. Calculate the intersection of each selection scheme. Calculate the union of the intersections of each selection scheme to obtain the intersection labeled with N-1. The process is repeated in a loop, counting the total number of data items in the intersection of labeled data. When the total number of data items reaches a preset threshold, a loop exit instruction is generated.

[0009] Furthermore, the step of constructing a matching processing scheme based on a globally shared database, as a pre-processing step for each built-in large model processor, includes: Share the global shared database to each processor of the built-in large model; Before the text is sent to the processor of the built-in large model for processing, the text similarity is calculated by traversing the entire shared database based on the text. When the text similarity reaches the preset similarity threshold, the processing process corresponding to the data item is read, and the processing process and text are sent to the processor of the built-in large model.

[0010] Furthermore, the step of periodically acquiring the processing speed and accuracy of the text task set, and updating the total number of data items in the global shared database based on the processing speed and accuracy includes: Monitor the text processing progress of the text task set and calculate the amount of text processed per unit time in real time as the processing speed; Every preset time interval, the task text that has undergone the matching processing scheme is marked, and traditional processing instructions pointing to the marked task text are generated; Read the text processing result corresponding to the matching processing scheme as the first result, and read the text processing result corresponding to the traditional processing instruction as the second result; Using the second result as the true result, calculate the processing accuracy of the first result; Adjustment commands are generated periodically. When an adjustment command is generated, the total number threshold is adjusted, the difference between the adjusted processing speed and the processing speed before adjustment is calculated, the total number threshold is determined based on the difference, and the processing accuracy is obtained. The adjusted processing speed is adjusted according to the processing accuracy, and the adjusted processing speed is directly proportional to the processing accuracy.

[0011] The present invention also provides a RAG multi-granularity recall system based on large model semantic enhancement, the system comprising: The text processing recording module receives a set of text tasks, processes the text in the set based on a built-in large-model processor, and records the processing progress synchronously. The private domain library construction module is used to compare the processing progress and processed text of any two processors with built-in large models, and construct a private shared database for the two processors with built-in large models; the shared database includes text items and processing process items; The global database construction module is used to compare all private domain shared databases to determine the global shared database; the total number of data items in the global shared database is a dynamic value. The preprocessing determination module is used to construct a matching processing scheme based on a globally shared database, serving as the preprocessing process for each built-in large model processor; The global database update module is used to periodically obtain the processing speed and accuracy of text task sets, and update the total number of data items in the global shared database based on the processing speed and accuracy.

[0012] Furthermore, the private domain library construction module includes: The work log query unit is used to query the work logs of any two built-in large model processors. The standardization processing unit is used to query the processing time of each text in the work log, standardize the processing time, and obtain a standard processing time; the standard processing time is a percentage. The secondary clustering unit is used to perform preliminary clustering of text based on standard time consumption, compare each group of text after preliminary clustering, and perform secondary clustering of text based on the comparison results; The feature text selection unit is used to select feature text for each class of text after secondary clustering, read the processing progress of the feature text, insert it into the database, insert the name tags of the processors of the two built-in large models into the database, and obtain the private domain shared database of the processors of the two built-in large models.

[0013] Furthermore, the global library construction module includes: The initial calculation unit reads the N private shared databases that have been created, compares the text items in the N private shared databases, calculates the intersection, and uses N as the label of the intersection; The intermediate computing unit is used to calculate the difference set between each private domain shared database and the intersection. It enumerates and selects N-1 difference sets from all the difference sets, calculates the intersection of each selection scheme, and calculates the union of the intersections of each selection scheme to obtain the intersection with N-1 as the label. The loop control unit is used to execute loops and count the total number of data items in the intersection of tagged data. When the total number of data items reaches a preset threshold, a loop exit command is generated.

[0014] Furthermore, the preceding process determination module includes: Shared execution units are used to share the global shared database to each processor of the built-in large model; The text comparison unit is used to calculate text similarity based on a full-domain shared database before sending the text to the processor of the built-in large model for processing; The data sending unit is used to read the processing process corresponding to the data item when the text similarity reaches a preset similarity threshold, and send the processing process and text to the processor of the built-in large model.

[0015] Compared with the prior art, the beneficial effects of the present invention are: When faced with similar tasks, this invention constructs a shared database containing text and processing processes based on historical data from all processors with built-in large models. Before processing the text, data items are matched in the shared database, and the corresponding processing processes are queried to obtain a very small range of recall results. Then, the processor directly applies the processing processes to process the text. Although the accuracy is slightly insufficient, the range obtained by the recall process is extremely small, which greatly improves the response speed with only a slight reduction in accuracy. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention.

[0017] Figure 1 The overall flowchart of the RAG multi-granularity recall method based on large model semantic enhancement is shown.

[0018] Figure 2 The first sub-flow diagram of the RAG multi-granularity recall method based on large model semantic enhancement is shown.

[0019] Figure 3 The second sub-flow diagram of the RAG multi-granularity recall method based on large model semantic enhancement is shown.

[0020] Figure 4 The third sub-flow diagram of the RAG multi-granularity recall method based on large model semantic enhancement is shown.

[0021] Figure 5 The fourth sub-flow diagram of the RAG multi-granularity recall method based on large model semantic enhancement is shown.

[0022] Figure 6 The structure diagram of the RAG multi-granularity recall system based on large model semantic enhancement is shown. Detailed Implementation

[0023] To make the technical problems, technical solutions, and beneficial effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0024] Figure 1 The above is a flowchart of the RAG multi-granularity recall method and system based on large model semantic enhancement. In this embodiment of the invention, a RAG multi-granularity recall method based on large model semantic enhancement is provided, the method comprising: Step S100: Receive the text task set, process the text in the text task set based on the built-in large model processor, and record the processing progress synchronously; The technical solution of this invention is applied to a processor cluster with a built-in large model. The data to be processed is text data, which belongs to the same text set. For example, all the texts that need to be processed for a certain service within a time period. These texts include multiple repeated scenarios and have a high degree of similarity. When processing, similar processing schemes can be adopted. Specifically, for example, in the scenario of sharing a batch of files, the file recipients have similar needs for the file. Different file recipients may have highly similar interactions. In layman's terms, the unclear parts may be similar.

[0025] Step S200: For any two processors with built-in large models, compare the processing progress and the processed text to construct a private shared database for the two processors with built-in large models; the shared database includes text items and processing progress items; The interaction with the user is completed by a processor cluster. Each processor is a processor with a built-in large model. The number of processors is not unique. By comparing their processing processes and the processed text (stored in the storage database corresponding to each processor), some similar texts can be selected. The processing processes corresponding to these similar texts are read, and data items including text and processing processes are created. All data items are counted to obtain a private shared database belonging to the processors of the two built-in large models.

[0026] It should be noted that the degree of text similarity is controlled by a threshold set in advance by the staff. In the technical solution of this invention, this threshold is very high, so that when the processing is applied to another text, there will not be too much incongruity.

[0027] Step S300: Compare all private domain shared databases to determine the global shared database; wherein, the total number of data items in the global shared database is a dynamic value; Each processor with a built-in large model has a private shared database between each other. If there are N processors with built-in large models, each processor with a built-in large model has a private shared database with the other N-1 processors with built-in large models. In practical applications, if these N-1 private shared databases are used every time, the text needs to be compared with the data in each of these private shared databases, resulting in a large number of comparisons, which will affect efficiency. It is better to send the text directly to the processor for processing. Therefore, the above content introduces a global database construction scheme based on private shared databases. Some data items are selected from each private shared database to form a database called the global shared database. It should be noted that the total number of data items in the global shared database is dynamic, that is, the process of selecting how many data items to form the global shared database is uncertain. In the technical solution of this invention, a data item determination scheme based on actual conditions is introduced, which is not a traditional fixed value. Of course, an initial value can be determined during the first construction, which is different from the existing fixed number of databases and is an innovation of the technical solution of this invention.

[0028] Step S400: Construct a matching processing scheme based on a globally shared database as a pre-processing process for each built-in large model processor; After the global shared database is constructed, it serves as the global database for all built-in large model processors. A matching processing scheme based on the global shared database is then built as a pre-processing step for each built-in large model processor. Specifically, for text task sets, the process involves continuously reading task text and assigning it to different built-in large model processors (the assignment process uses existing methods, such as parallel assignment). After the global shared database is constructed, before sending the text to the built-in large model processors, the system traverses the global shared database to match data items. Each data item includes the text and the processing step. The processing steps within the matched data items are then read. The processing flow involves sending the text to the processor of the built-in large model simultaneously with the read processing flow. When the processor of the built-in large model receives the text and the processing flow, it determines a "text processing range" based on the processing flow and then processes the text. At this point, the text recognition process is omitted. Although the matching accuracy may decrease slightly, this shortcoming can be ignored if the text is sufficiently similar. In fact, determining the "text processing range" means limiting a small model application range during the large model processing. This can be understood as the "context" of the interaction process, which is itself a kind of "recall" concept.

[0029] Step S500: Periodically obtain the processing speed and accuracy of the text task set, and update the total number of data items in the global shared database based on the processing speed and accuracy; Step S500 is a recursive adjustment process. Its focus is on adjusting the total number of data items in the global shared database in step S300. Specifically, a listening port is set in the text task set to periodically obtain the processing speed and processing accuracy of the text task set. The processing speed is the number of texts processed per unit time, and the processing accuracy is obtained by comparing the results of using a preprocessing process and not using a preprocessing process. The total number of data items in the global shared database is updated according to the processing speed and processing accuracy.

[0030] It should be noted that the relationship between the total number of data items and processing speed is actually analogous to that of gradient descent. In a globally shared database, the number of data items starts from zero. The more data items there are, the faster the processing speed. However, as the total number of data items continues to increase, the traversal process may take longer, which will reduce the processing speed. Therefore, there is a peak in processing speed. Using the idea of ​​gradient descent, the total number of data items can be determined by the processing speed. Regarding processing accuracy, it can affect the processing speed, and thus the total number of data items. The lower the processing accuracy, the slower the processing speed is considered. This adjustment can be achieved through a correction coefficient.

[0031] Figure 2 The first sub-flow diagram of the RAG multi-granularity recall method based on large model semantic enhancement is shown. The step of constructing a private shared database of the processors of any two built-in large models by comparing the processing progress and the processed text includes: For any two processors with built-in large models, query the working logs of the processors with built-in large models; The processing time for each text file is queried from the work log. The processing time is then standardized to obtain the standard processing time, which is expressed as a percentage. The texts are initially clustered based on standard time consumption. Each cluster of texts is then compared pairwise, and a second clustering is performed based on the comparison results. For each class of text after secondary clustering, select feature text, read the processing progress of feature text, insert it into the database, insert the name labels of the processors of the two built-in large models into the database, and obtain the private domain shared database of the processors of the two built-in large models.

[0032] In one example of the technical solution of this invention, the construction process of the private domain shared database is described. For any two processors with built-in large models, the work logs of the processors with built-in large models are queried. The processing time of each text is queried in the work logs. The processing time is standardized to obtain the standard processing time. The standardization process is to calculate the ratio of the processing time to the preset maximum processing time. The purpose of standardization is to eliminate the influence of the processors with built-in large models to a certain extent. For example, the processing time of the processor with better performance is inherently shorter. The standardization process converts the processing time into a relative scale, which is mainly related to the complexity of the text itself. Of course, if the performance parameters of all processors with built-in large models are the same, this conversion process can be omitted.

[0033] The text is initially clustered based on the standard time consumption. That is, the texts with sufficiently small differences in standard time consumption are first clustered into one class. This process only requires comparing the time consumption parameter, which is very fast. On this basis, each class of texts after the initial clustering is compared pairwise, and the texts are clustered a second time based on the comparison results. This process is a text comparison process, which requires more resources, but the comparison results are more accurate. Through two-step clustering, a clustering result with both high speed and accuracy is obtained.

[0034] After clustering, for each class of text after secondary clustering, feature text is selected, the processing progress of the feature text is read, and it is inserted into the database. The name labels of the processors of the two built-in large models are inserted into the database to obtain the private domain shared database of the processors of the two built-in large models.

[0035] In the above content, during the initial clustering of texts based on standard time consumption, the clustering distance is the absolute difference of standard time consumption; For each text in the initial clustering, pairwise comparisons are performed. During the secondary clustering process based on the comparison results, the clustering distance is inversely proportional to the text similarity. The clustering condition is that the maximum value of the pairwise distance between any two texts in the clustering result is less than a preset distance threshold. The above content explains the details of the two-step clustering process. The clustering process adopts existing clustering schemes. Regardless of the clustering scheme, it will involve parameters that describe the degree of difference in data. The above content is distance. Among them, text similarity can be achieved using conventional text comparison techniques. The condition in the secondary clustering process is that the maximum value of the distance between any two texts in the clustering result is less than a preset distance threshold. This is a very strict condition. It is not enough for two texts to be classified into the same class if they are similar enough. It also needs to satisfy that the two texts are sufficiently similar to the texts in their respective classes in order to be classified into the same class. Otherwise, they belong to a new class. Based on this, the clustering scheme actually adopts a clustering scheme with no limit on the number of classes.

[0036] Furthermore, the process of selecting feature text is as follows: Read each text class after the second clustering in sequence. Read the text in that class in sequence, calculate the mean distance of the text to all other texts, and select the text with the smallest mean distance as the feature text.

[0037] The process of selecting feature text involves choosing a text as a representative from a class of texts, sequentially reading each class of texts after secondary clustering, sequentially reading texts within that class, calculating the mean distance between it and all other texts, and selecting the text with the smallest mean distance as the feature text.

[0038] Figure 3 The second sub-process flowchart of the RAG multi-granularity recall method based on large model semantic enhancement is shown. The step of comparing all private domain shared databases to determine the global domain shared database includes: Read the N private shared databases that have been created, compare the text items in the N private shared databases, calculate the intersection, and use N as the label of the intersection; Calculate the difference between each private shared database and the intersection. Enumerate and select N-1 difference sets from all the difference sets. Calculate the intersection of each selection scheme. Calculate the union of the intersections of each selection scheme to obtain the intersection labeled with N-1. The process is repeated in a loop, counting the total number of data items in the intersection of labeled data. When the total number of data items reaches a preset threshold, a loop exit instruction is generated.

[0039] The above content describes the integration process of private shared databases. It involves reading N created private shared databases, comparing the text items in the N private shared databases, and calculating the intersection. The scheme used in the text comparison process can be the same as the scheme used in the clustering process described above, and existing text comparison techniques can be used.

[0040] The implicit meaning of the N private shared databases already created is that there are a total of N private shared databases. The data in the intersection obtained at this time has appeared in the historical data of all processors with built-in large models. Then, the intersection part is removed from each private shared database, and the data is reduced further. N-1 databases are selected from the N private shared databases, and the intersection is calculated again. At this time, the data in these intersections has appeared in the historical data of N-1 processors with built-in large models. And so on, intersections corresponding to different numbers of labels can be obtained, with labels including N, N-1, N-2, etc.

[0041] Because there are many ways to select data, the number of random labels decreases, and the data in the intersection becomes more and more limited. However, the total number of data in the output intersection will increase. When the total number of data items reaches the preset total threshold (when there are enough), a loop exit instruction is generated to stop the loop.

[0042] Figure 4 The flowchart of the third sub-process of the RAG multi-granularity recall method based on large model semantic enhancement is shown. The steps of constructing a matching processing scheme based on a global shared database, which serves as a pre-processing process for each built-in large model, include: Share the global shared database to each processor of the built-in large model; Before the text is sent to the processor of the built-in large model for processing, the text similarity is calculated by traversing the entire shared database based on the text. When the text similarity reaches the preset similarity threshold, the processing process corresponding to the data item is read, and the processing process and text are sent to the processor of the built-in large model.

[0043] In one example of the technical solution of this invention, the pre-processing process of the processor of the built-in large model is described. Based on the existing technology of text comparison, this process is actually not complicated. The global shared database is shared to each processor of the built-in large model. The sharing method is to establish a connection channel. Before the text is sent to the processor of the built-in large model for processing, the global shared database is traversed based on the text to calculate the text similarity. When the text similarity reaches the preset similarity threshold, the corresponding data item is taken as the matching data item, called the target data item. The processing process corresponding to the data item is read, and the processing process and text are sent to the processor of the built-in large model.

[0044] Figure 5 The fourth sub-process flowchart of the RAG multi-granularity recall method based on large model semantic enhancement is shown. The step of periodically obtaining the processing speed and accuracy of the text task set, and updating the total number of data items in the global shared database according to the processing speed and accuracy includes: Monitor the text processing progress of the text task set and calculate the amount of text processed per unit time in real time as the processing speed; Every preset time interval, the task text that has undergone the matching processing scheme is marked, and traditional processing instructions pointing to the marked task text are generated; Read the text processing result corresponding to the matching processing scheme as the first result, and read the text processing result corresponding to the traditional processing instruction as the second result; Using the second result as the true result, calculate the processing accuracy of the first result; Adjustment commands are generated periodically. When an adjustment command is generated, the total number threshold is adjusted, the difference between the adjusted processing speed and the processing speed before adjustment is calculated, the total number threshold is determined based on the difference, and the processing accuracy is obtained. The adjusted processing speed is adjusted according to the processing accuracy, and the adjusted processing speed is directly proportional to the processing accuracy.

[0045] In one example of the technical solution of this invention, the recursive adjustment process is described. The text processing progress of the text task set is monitored, and the number of texts processed per unit time is calculated in real time as the processing speed. Every preset time interval, the task texts that have undergone the matching processing scheme are marked, and traditional processing instructions pointing to the marked task texts are generated. The traditional processing instructions are used to control the processor of the built-in large model to perform the original processing. The text processing result corresponding to the matching processing scheme is read as the first result, and the text processing result corresponding to the traditional processing instructions is read as the second result. The first result and the second result are compared, and the difference is calculated. The difference is actually the error rate. The processing accuracy is determined according to the inverse proportion of the error rate.

[0046] Under the condition that the processing speed and processing accuracy are known, adjustment instructions are generated periodically. When an adjustment instruction is generated, the total threshold is adjusted, and the difference between the processing speed after adjustment and the processing speed before adjustment is calculated. The total threshold is determined based on the difference. If the processing speed increases, the difference is positive, and the total threshold increases. The magnitude of the increase in the total threshold is determined by the absolute value of the difference. Conversely, if the processing speed decreases, the difference is negative, and the total threshold decreases. The magnitude of the decrease in the total threshold is determined by the absolute value of the difference.

[0047] Furthermore, when the processing accuracy is obtained, the adjusted processing speed is adjusted according to the processing accuracy. The adjusted processing speed is directly proportional to the processing accuracy. That is, the higher the processing accuracy, the higher the adjusted processing speed, and correspondingly, the processing speed will become larger, and the total threshold will increase by a greater margin.

[0048] Of course, the relationship between processing accuracy and processing speed can be determined by the staff based on the actual situation. The relationship between processing speed and the total threshold is based on the gradient descent method. Processing accuracy only needs to establish a mapping relationship with processing speed, without needing to introduce the relationship between processing accuracy and the total threshold. This makes the adjustment logic clearer.

[0049] Figure 6 A structural diagram of a RAG multi-granularity recall system based on large model semantic enhancement is shown. In a preferred embodiment of the technical solution of the present invention, a RAG multi-granularity recall system based on large model semantic enhancement is also provided. The system 10 includes: The text processing recording module 11 is used to receive a set of text tasks, process the text in the set of text tasks based on the processor with the built-in large model, and record the processing progress synchronously. The private domain library construction module 12 is used to compare the processing progress and processed text of any two processors with built-in large models, and construct a private domain shared database for the two processors with built-in large models; the shared database includes text items and processing process items; The global database construction module 13 is used to compare all private domain shared databases to determine the global shared database; the total number of data items in the global shared database is a dynamic value. The preprocessing determination module 14 is used to construct a matching processing scheme based on a global shared database, serving as the preprocessing process for each built-in large model processor; The global database update module 15 is used to periodically obtain the processing speed and accuracy of the text task set, and update the total number of data items in the global shared database according to the processing speed and accuracy.

[0050] Furthermore, the private domain library construction module 12 includes: The work log query unit is used to query the work logs of any two built-in large model processors. The standardization processing unit is used to query the processing time of each text in the work log, standardize the processing time, and obtain a standard processing time; the standard processing time is a percentage. The secondary clustering unit is used to perform preliminary clustering of text based on standard time consumption, compare each group of text after preliminary clustering, and perform secondary clustering of text based on the comparison results; The feature text selection unit is used to select feature text for each class of text after secondary clustering, read the processing progress of the feature text, insert it into the database, insert the name tags of the processors of the two built-in large models into the database, and obtain the private domain shared database of the processors of the two built-in large models.

[0051] Specifically, the global library construction module 13 includes: The initial calculation unit reads the N private shared databases that have been created, compares the text items in the N private shared databases, calculates the intersection, and uses N as the label of the intersection; The intermediate computing unit is used to calculate the difference set between each private domain shared database and the intersection. It enumerates and selects N-1 difference sets from all the difference sets, calculates the intersection of each selection scheme, and calculates the union of the intersections of each selection scheme to obtain the intersection with N-1 as the label. The loop control unit is used to execute loops and count the total number of data items in the intersection of tagged data. When the total number of data items reaches a preset threshold, a loop exit command is generated.

[0052] Furthermore, the preprocess determination module 14 includes: Shared execution units are used to share the global shared database to each processor of the built-in large model; The text comparison unit is used to calculate text similarity based on a full-domain shared database before sending the text to the processor of the built-in large model for processing; The data sending unit is used to read the processing process corresponding to the data item when the text similarity reaches a preset similarity threshold, and send the processing process and text to the processor of the built-in large model.

[0053] The above are merely preferred embodiments of the present invention and do not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A multi-granularity recall method for RAG based on large model semantic enhancement, characterized in that... The method includes: It receives a set of text tasks, processes the text in the set based on a processor with a built-in large model, and records the processing progress synchronously. For any two processors with built-in large models, compare the processing progress and the text already processed to construct a private shared database for the two processors with built-in large models; the shared database includes text items and processing progress items; By comparing all private domain shared databases, the global shared database is determined; the total number of data items in the global shared database is a dynamic value. A matching-based processing scheme based on a globally shared database is constructed as a pre-processing step for each built-in large model processor; The processing speed and accuracy of the text task set are periodically obtained, and the total number of data items in the global shared database is updated based on the processing speed and accuracy.

2. The RAG multi-granularity recall method based on large model semantic enhancement according to claim 1, characterized in that... The step of constructing a private shared database for any two processors with built-in large models by comparing their processing progress and the text they have processed includes: For any two processors with built-in large models, query the working logs of the processors with built-in large models; The processing time for each text file is queried from the work log. The processing time is then standardized to obtain the standard processing time, which is expressed as a percentage. The texts are initially clustered based on standard time consumption. Each cluster of texts is then compared pairwise, and a second clustering is performed based on the comparison results. For each class of text after secondary clustering, select feature text, read the processing progress of feature text, insert it into the database, insert the name labels of the processors of the two built-in large models into the database, and obtain the private domain shared database of the processors of the two built-in large models.

3. The RAG multi-granularity recall method based on large model semantic enhancement according to claim 2, characterized in that... In the initial clustering of texts based on standard time consumption, the cluster distance is the absolute difference of standard time consumption; For each text in the initial clustering, pairwise comparisons are performed. During the secondary clustering process based on the comparison results, the clustering distance is inversely proportional to the text similarity. The clustering condition is that the maximum value of the pairwise distance between any two texts in the clustering result is less than a preset distance threshold. The process of selecting feature text is as follows: Read each text class after the second clustering in sequence. Read the text in that class in sequence, calculate the mean distance of the text to all other texts, and select the text with the smallest mean distance as the feature text.

4. The RAG multi-granularity recall method based on large model semantic enhancement according to claim 1, characterized in that... The step of comparing all private domain shared databases to determine the global domain shared database includes: Read the N private shared databases that have been created, compare the text items in the N private shared databases, calculate the intersection, and use N as the label of the intersection; Calculate the difference between each private shared database and the intersection. Enumerate and select N-1 difference sets from all the difference sets. Calculate the intersection of each selection scheme. Calculate the union of the intersections of each selection scheme to obtain the intersection labeled with N-1. The process is repeated in a loop, counting the total number of data items in the intersection of labeled data. When the total number of data items reaches a preset threshold, a loop exit instruction is generated.

5. The RAG multi-granularity recall method based on large model semantic enhancement according to claim 1, characterized in that... The steps for constructing a matching processing scheme based on a globally shared database, serving as a pre-processing step for each built-in large model processor, include: Share the global shared database to each processor of the built-in large model; Before the text is sent to the processor of the built-in large model for processing, the text similarity is calculated by traversing the entire shared database based on the text. When the text similarity reaches the preset similarity threshold, the processing process corresponding to the data item is read, and the processing process and text are sent to the processor of the built-in large model.

6. The RAG multi-granularity recall method based on large model semantic enhancement according to claim 4, characterized in that... The step of periodically acquiring the processing speed and accuracy of the text task set, and updating the total number of data items in the global shared database based on the processing speed and accuracy includes: Monitor the text processing progress of the text task set and calculate the amount of text processed per unit time in real time as the processing speed; Every preset time interval, the task text that has undergone the matching processing scheme is marked, and traditional processing instructions pointing to the marked task text are generated; Read the text processing result corresponding to the matching processing scheme as the first result, and read the text processing result corresponding to the traditional processing instruction as the second result; Using the second result as the true result, calculate the processing accuracy of the first result; Adjustment commands are generated periodically. When an adjustment command is generated, the total number threshold is adjusted, the difference between the adjusted processing speed and the processing speed before adjustment is calculated, the total number threshold is determined based on the difference, and the processing accuracy is obtained. The adjusted processing speed is adjusted according to the processing accuracy, and the adjusted processing speed is directly proportional to the processing accuracy.

7. A RAG multi-granularity recall system based on large model semantic enhancement, characterized in that... The system includes: The text processing recording module receives a set of text tasks, processes the text in the set based on a built-in large-model processor, and records the processing progress synchronously. The private domain library construction module is used to compare the processing progress and processed text of any two processors with built-in large models, and construct a private shared database for the two processors with built-in large models; the shared database includes text items and processing process items; The global database construction module is used to compare all private domain shared databases to determine the global shared database; the total number of data items in the global shared database is a dynamic value. The preprocessing determination module is used to construct a matching processing scheme based on a globally shared database, serving as the preprocessing process for each built-in large model processor; The global database update module is used to periodically obtain the processing speed and accuracy of text task sets, and update the total number of data items in the global shared database based on the processing speed and accuracy.

8. The RAG multi-granularity recall system based on large model semantic enhancement according to claim 7, characterized in that... The private domain library construction module includes: The work log query unit is used to query the work logs of any two built-in large model processors. The standardization processing unit is used to query the processing time of each text in the work log, standardize the processing time, and obtain a standard processing time; the standard processing time is a percentage. The secondary clustering unit is used to perform preliminary clustering of text based on standard time consumption, compare each group of text after preliminary clustering, and perform secondary clustering of text based on the comparison results; The feature text selection unit is used to select feature text for each class of text after secondary clustering, read the processing progress of the feature text, insert it into the database, insert the name tags of the processors of the two built-in large models into the database, and obtain the private domain shared database of the processors of the two built-in large models.

9. The RAG multi-granularity recall system based on large model semantic enhancement according to claim 7, characterized in that... The global library construction module includes: The initial calculation unit is used to read the N private shared databases that have been created, compare the text items in the N private shared databases, calculate the intersection, and use N as the label of the intersection; The intermediate computing unit is used to calculate the difference set between each private domain shared database and the intersection. It enumerates and selects N-1 difference sets from all the difference sets, calculates the intersection of each selection scheme, and calculates the union of the intersections of each selection scheme to obtain the intersection with N-1 as the label. The loop control unit is used to execute loops and count the total number of data items in the intersection of tagged data. When the total number of data items reaches a preset threshold, a loop exit command is generated.

10. The RAG multi-granularity recall system based on large model semantic enhancement according to claim 7, characterized in that... The preceding process determination module includes: Shared execution units are used to share the global shared database to each processor of the built-in large model; The text comparison unit is used to calculate text similarity based on a full-domain shared database before sending the text to the processor of the built-in large model for processing; The data sending unit is used to read the processing process corresponding to the data item when the text similarity reaches a preset similarity threshold, and send the processing process and text to the processor of the built-in large model.

Images