Method and device for executing reasoning task of LLM, equipment cluster and product
By using paragraph summaries and their relevance to the inference task to filter paragraphs, the computational resource consumption and time consumption issues of LLM under long text data are solved, improving the execution efficiency of inference tasks and user experience.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HUAWEI TECH CO LTD
- Filing Date
- 2024-11-22
- Publication Date
- 2026-05-22
AI Technical Summary
Existing LLMs consume significant computational resources and are time-consuming when processing inference tasks involving long text data, resulting in excessively long user wait times and negatively impacting user experience.
By using paragraph summaries to filter target paragraphs based on their relevance to the reasoning task, the calculation of the relevance of each paragraph to the reasoning task is reduced, saving computational resources and shortening the filtering time.
It reduces memory usage, increases throughput, shortens the total time of the end-to-end process, and improves the inference efficiency of LLM on long text data.
Smart Images
Figure CN122072841A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence (AI) technology, and in particular to a method, apparatus, device cluster and product for performing reasoning tasks of a large language model (LLM). Background Technology
[0002] LLM-based AI applications can leverage the inference capabilities of LLM to solve various real-world reasoning tasks, such as long text generation and question answering. The LLM inference process can be summarized as follows: LLM acquires the user-requested inference task and the text data required to execute it, performs inference based on the text data, and obtains the inference result. For example, if the inference task is a question-and-answer task, LLM can combine text data to perform inference and output the answer. Improving the execution efficiency of inference tasks in scenarios where the text data consists of long documents is a current research hotspot. Summary of the Invention
[0003] This application provides a method, apparatus, device cluster, and product for executing LLM inference tasks, which can improve the execution efficiency of LLM inference tasks. The technical solution is as follows:
[0004] Firstly, a method for executing an LLM inference task is provided. In this method, an inference task and text data are acquired, the text data including multiple paragraphs; the relevance of each paragraph to the inference task is determined based on the relevance of the paragraph summary to the inference task, the content of the paragraph summary being a portion of the content extracted from each paragraph; based on the relevance of each paragraph to the inference task, a target number of paragraphs are determined from the multiple paragraphs; the target number of paragraphs and the inference task are input into the LLM, so that the LLM outputs the inference result of the inference task based on the target number of paragraphs.
[0005] This method utilizes the relevance of each paragraph's summary to the inference task to determine the relevance of each paragraph to the task. Then, a target number of paragraphs are selected from multiple paragraphs to determine the inference result. Since the paragraph summary is a partial extraction from each paragraph, it eliminates the need to calculate the relevance of the entire content of each paragraph to the inference task. This saves computational resources, reducing memory usage and improving throughput. Furthermore, it shortens the time required to filter the target number of paragraphs from text data, thereby reducing the overall end-to-end process time and preventing users from waiting excessively for answers after initiating a question-and-answer task.
[0006] Based on the method provided in the first aspect, in one possible implementation, the paragraph summary of each paragraph is the paragraph title or paragraph summary of each paragraph, and the paragraph summary is the paragraph after each paragraph has been shortened.
[0007] For any given paragraph, its summary can be understood as a portion of the content extracted from that paragraph, representing its core message. For example, the summary could be the paragraph title or a summary of the paragraph. The summary refers to the paragraph after it has been abridged. Therefore, for any given paragraph, the text size of its summary is smaller than the text size of the paragraph itself. Consequently, the computational resources required to determine the relevance of the summary to the reasoning task are less than those required to determine the relevance of the paragraph itself to the reasoning task. Correspondingly, the time required to determine the relevance of the summary to the reasoning task is less than the time required to determine the relevance of the paragraph itself to the reasoning task.
[0008] Based on the method provided in the first aspect, in one possible implementation, the process of determining the relevance of each paragraph to the reasoning task according to the relevance of the paragraph summary to the reasoning task can be as follows: For the first paragraph among multiple paragraphs, determine whether the relevance of the paragraph title of the first paragraph to the reasoning task is greater than a first threshold; when the relevance of the paragraph title of the first paragraph to the reasoning task exceeds the first threshold, use the relevance of the paragraph title of the first paragraph to the reasoning task as the relevance of the first paragraph to the reasoning task; when the relevance of the paragraph title of the first paragraph to the reasoning task is lower than the first threshold, determine the relevance of the paragraph summary of the first paragraph to the reasoning task, and when the relevance of the paragraph summary of the first paragraph to the reasoning task exceeds a second threshold, use the relevance of the paragraph summary of the first paragraph to the reasoning task as the relevance of the first paragraph to the reasoning task.
[0009] That is, in this embodiment of the application, when the relevance of the paragraph title of the first paragraph to the reasoning task exceeds a first threshold, the paragraph title of the first paragraph is used as the paragraph summary of the first paragraph. When the relevance of the paragraph title of the first paragraph to the reasoning task is lower than the first threshold, the paragraph summary of the first paragraph is used to determine the paragraph summary of the first paragraph.
[0010] Based on the method provided in the first aspect, in one possible implementation, after determining the relevance of the paragraph summary of the first paragraph to the reasoning task, the first paragraph is discarded when the relevance of the paragraph summary of the first paragraph to the reasoning task is lower than a second threshold.
[0011] If the relevance of the paragraph summary of the first paragraph to the reasoning task is lower than the second threshold, it indicates that the correlation between the paragraph summary of the first paragraph and the reasoning task is relatively weak, and therefore the first paragraph can be discarded. Discarding the first paragraph can be understood as: the first paragraph is not considered as a candidate paragraph for the subsequent selection of the target number of paragraphs.
[0012] Based on the method provided in the first aspect, in one possible implementation, when the relevance of the paragraph title of the first paragraph to the inference task is lower than a first threshold, the process of determining the relevance of the paragraph summary of the first paragraph to the inference task can be as follows: when the relevance of the paragraph title of the first paragraph to the inference task is lower than the first threshold but higher than a third threshold, the relevance of the paragraph summary of the first paragraph to the inference task is determined. Accordingly, when the relevance of the paragraph title of the first paragraph to the inference task is lower than the third threshold, the first paragraph is discarded.
[0013] In this embodiment, when the relevance between the paragraph title of the first paragraph and the inference task is too low, it is possible that all content in the first paragraph has a low relevance to the inference task. In this scenario, it is unnecessary to calculate the relevance between the paragraph summary of the first paragraph and the inference task. Therefore, this embodiment also sets a third threshold. Thus, only when the relevance between the paragraph title of the first paragraph and the inference task falls between the first threshold and the third threshold will the relevance between the paragraph summary of the first paragraph and the inference task be further calculated; otherwise, the first paragraph is discarded.
[0014] Based on the method provided in the first aspect, in one possible implementation, for the first paragraph among multiple paragraphs, the perplexity level (PPL) of each word in the multiple words included in the first paragraph is determined; words with PPL less than the PPL threshold are deleted from the first paragraph to obtain a paragraph summary of the first paragraph.
[0015] The PPL (Programmatical Point of Interest) for each lexical unit indicates the degree of association between that lexical unit and other lexical units in the first paragraph. The PPL threshold is a pre-set threshold. For any lexical unit in the PPL, if its PPL is lower than the threshold, it is considered redundant and can be deleted. Conversely, if its PPL exceeds the threshold, it is considered non-redundant and is retained.
[0016] Based on the method provided in the first aspect, in one possible implementation, the reasoning task is a question-and-answer task, and the length of the text data exceeds the reference length.
[0017] In this embodiment, the inference task can be a question-and-answer task, such as a question. The length of the text data can exceed the reference length, meaning the text data is long text data. In other words, the method provided in this embodiment can be applied to scenarios where answers to question-and-answer tasks are obtained through long text data. For example, in long document question-and-answer applications, using the method provided in this embodiment to obtain answers to user-initiated questions can not only avoid wasting computational resources during the answer acquisition process of long document question-and-answer applications, but also shorten the total time of the end-to-end process, preventing users from having to wait a long time to get answers after initiating a question.
[0018] Secondly, an execution apparatus for an LLM inference task is provided, the execution apparatus for an LLM inference task having the function of implementing the behavior of the execution apparatus method for an LLM inference task in the first aspect described above. The execution apparatus for an LLM inference task includes at least one module for implementing the execution apparatus method for an LLM inference task provided in the first aspect.
[0019] Thirdly, a computing device cluster is provided, including at least one computing device, each computing device including a processor and memory;
[0020] The processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the cluster of computing devices performs the execution method of the LLM inference task provided in the first aspect above.
[0021] Fourthly, a computer program product containing instructions is provided, which, when run by a cluster of computing devices, causes the cluster of computing devices to execute the LLM inference task execution method provided in the first aspect.
[0022] Fifthly, a computer-readable storage medium is provided, including computer program instructions, which, when executed by a cluster of computing devices, enable the cluster of computing devices to execute the LLM inference task execution method provided in the first aspect.
[0023] The technical effects achieved by the corresponding technical means in the second to fifth aspects mentioned above are similar, and will not be repeated here. Attached Figure Description
[0024] Figure 1 This is a flowchart illustrating the process of performing question-answering tasks using an LLM in a long document question-answering application.
[0025] Figure 2 This is a schematic diagram of the architecture of an execution system for an LLM inference task provided in an embodiment of this application;
[0026] Figure 3 This is a flowchart of an LLM inference task execution method provided in an embodiment of this application;
[0027] Figure 4 This is a schematic diagram illustrating the correspondence between paragraphs and paragraph titles provided in an embodiment of this application;
[0028] Figure 5 This is a flowchart illustrating a method for trimming the first paragraph, as provided in an embodiment of this application.
[0029] Figure 6 This is a flowchart illustrating a reasoning question-answering task provided in an embodiment of this application;
[0030] Figure 7 This is a schematic diagram of the structure of an execution device for an LLM inference task provided in an embodiment of this application;
[0031] Figure 8 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Detailed Implementation
[0032] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.
[0033] Before explaining the embodiments of this application, the application scenarios of the embodiments of this application will be introduced first.
[0034] Currently, most training data used to train LLMs consists of general knowledge, meaning there's a lack of domain-specific data. This leads to the following problem: when LLMs need to perform domain-specific reasoning tasks, they may not accurately understand and process domain-specific terminology, thus affecting their effectiveness in those areas. Therefore, when performing domain-specific reasoning tasks, additional contextual information can be input into the LLM, such as domain-specific knowledge. This can improve the effectiveness of LLMs in those areas, for example, by enhancing the quality of answers in question-answering scenarios, enabling LLMs to play a role in enterprise knowledge management, educational research, network operations, and other professional fields.
[0035] One application of this method is long document question answering. In long document question answering applications, text data and question-answering tasks are input into the LLM (Long Document Question Answering) system. The LLM uses the text data as contextual information and then infers the answer to the question-answering task based on the contextual information, eliminating the need for manual reading of the text data one by one to make a solution.
[0036] However, long document question answering currently faces two challenges: First, extremely long text data will exceed the context window limit of LLM, thus exceeding the processing capacity of LLM and causing a significant decrease in the inference performance of LLM; second, as the length of text data increases, the inference cost of LLM increases significantly, resulting in problems such as increased memory usage, reduced throughput, and increased platform load.
[0037] To address the issues encountered in long document question-answering applications, after inputting the question-answering task and text data, the application first splits the text data into paragraphs. Then, using a remanker model or vector library retrieval model, it identifies the paragraphs most relevant to the question-answering task from the split paragraphs, thus selecting the top-k paragraphs. These top-k paragraphs can be understood as the k paragraphs with the highest relevance to the question-answering task among multiple paragraphs. The top-k paragraphs and the question-answering task are then reassembled into prompts, which are then input into the LLM (Long Message Model). The LLM then infers the answer to the question-answering task based on the prompts, eliminating the need to input the entire text data into the LLM.
[0038] Figure 1 This is a flowchart illustrating the process of performing question-answering tasks using an LLM in a long document question-answering application. For example... Figure 1 As shown, after obtaining the text data and the question (i.e., the question-answering task), the text data is first split into multiple paragraphs. Figure 1 The process involves labeling multiple paragraphs as para_1, para_2, para_3, para_4, ..., para_n. Then, a rearrangement model is used to select the top-k paragraphs from these paragraphs, based on the question. These top-k paragraphs are then used as filtered text data. This filtered text data, along with the question, is input into the LLM (Limited Language Model) as prompts, which infers the answer to the question based on these prompts.
[0039] but Figure 1 The process shown has the following drawbacks:
[0040] (1) High resource consumption.
[0041] Figure 1 The process shown calculates the relevance of each paragraph to the question during the selection of the top-k paragraphs. However, the text data may contain a large amount of redundant text unrelated to the question. Calculating the relevance of this redundant text to the question will consume computational resources, resulting in wasted resources. Therefore, Figure 1 The illustrated process only reduces the amount of text input to the LLM; it doesn't actually save on resource consumption in the end-to-end process. An end-to-end process can be understood as starting with the user inputting a question and text data, and ending with the LLM outputting the answer to the question.
[0042] (2) Filter out the time taken for the top-k paragraphs.
[0043] because Figure 1 The process shown calculates the relevance of each paragraph to the question, which results in a long time to select the top-k paragraphs through the reordering model. This leads to a long total time for the end-to-end process, which in turn causes users to wait a long time to get an answer after submitting a question, affecting the user experience of using long document question and answer applications.
[0044] Based on this, this application provides a method for executing an LLM inference task. In this method, the relevance of each paragraph to the inference task is determined by utilizing the paragraph summary of each paragraph and its relevance to the inference task. Then, a target number of paragraphs are selected from multiple paragraphs to determine the inference result of the inference task. Since the content of the paragraph summary is a portion extracted from each paragraph, it is unnecessary to calculate the relevance of the entire content of each paragraph to the inference task. This saves computational resources, thereby reducing memory usage and improving throughput. Furthermore, it shortens the time required to filter out the target number of paragraphs from text data, thus reducing the total time of the end-to-end process and preventing users from having to wait a long time for an answer after initiating a question-and-answer task.
[0045] The following explains the execution system, method, and related products for LLM inference tasks provided in the embodiments of this application.
[0046] Figure 2 This is a schematic diagram of the architecture of an execution system for an LLM inference task provided in an embodiment of this application. Figure 2 As shown, the execution system 00 for the inference task of this LLM includes a paragraph filtering module 01 and a task inference module 02.
[0047] The paragraph filtering module 01 is used to filter a target number of paragraphs from the text data based on the method provided in this application embodiment. For example, if the target number is k, the paragraph filtering module 01 is used to filter the top-k paragraphs from the text data based on the method provided in this application embodiment. The task reasoning module 02 is used to input the filtered target number of paragraphs and the reasoning task into the LLM, and then the LLM outputs the reasoning result of the reasoning task.
[0048] In the process of filtering a target number of paragraphs from text data, the paragraph filtering module 01 only needs to calculate the relevance of each paragraph's summary to the inference task, without needing to calculate the relevance of each paragraph itself to the inference task. For any given paragraph, the paragraph summary can be understood as a portion of the content extracted from that paragraph, representing its basic content. For example, the paragraph summary could be the paragraph title or paragraph summary. The paragraph summary refers to the paragraph after it has been abridged.
[0049] Therefore, for any given paragraph, the text size of its summary is smaller than the text size of the paragraph itself. Consequently, the computational resources required to determine the relevance of the paragraph summary to the reasoning task are less than those required to determine the relevance of the paragraph itself to the reasoning task. Correspondingly, the time spent determining the relevance of the paragraph summary to the reasoning task is less than the time spent determining the relevance of the paragraph itself to the reasoning task.
[0050] Therefore, through Figure 2 The paragraph filtering module 01 shown can not only save the computing resources required to filter the top-k paragraphs, but also save the time required to filter the top-k paragraphs.
[0051] The detailed functions of the paragraph filtering module 01 and the task reasoning module 02 will be described in subsequent method implementation examples, and will not be elaborated here.
[0052] The paragraph filtering module 01 and the question-answering task reasoning module 02 are both software modules. These software modules can be deployed centrally or distributed on one or more computing devices to implement the LLM reasoning task execution method provided in this application embodiment.
[0053] in addition, Figure 2 This is an example of a system architecture for implementing the LLM inference task execution method provided in the embodiments of this application. In the embodiments of this application, the execution system for the LLM inference task may include more or fewer modules, which will not be described and illustrated here.
[0054] Figure 3 This is a flowchart illustrating an execution method for an LLM inference task provided in an embodiment of this application. Figure 3 As shown, the method includes the following steps.
[0055] Step 301: Obtain the reasoning task and text data, which includes multiple paragraphs.
[0056] In some embodiments, the acquisition of inference tasks and text data can be achieved by: displaying a task initiation interface, which includes a task input window and a context information input window; acquiring the inference task initiated by the user in response to information entered by the user in the task input window; and acquiring the text data entered by the user in response to information entered by the user in the context information input window.
[0057] After obtaining the text data, it can be further analyzed to split it into multiple paragraphs. In some embodiments, third-party tools such as Python can be used to split the text data into multiple paragraphs; however, this application does not limit the method of splitting the text data.
[0058] Furthermore, in this embodiment of the application, after splitting the text data into multiple paragraphs, the paragraph title of each paragraph can be further determined, and an index from the paragraph title of each paragraph to the corresponding paragraph can be established, so as to facilitate the subsequent determination of the relevance between the paragraph and the inference task based on the paragraph title.
[0059] Figure 4 This is a schematic diagram illustrating the correspondence between paragraphs and paragraph titles provided in an embodiment of this application. For example... Figure 4 As shown, the text data is split into multiple paragraphs, including paragraph 1, paragraph 2, and paragraph 3. The title of paragraph 1 is: "Differences between B-ultrasound, X-ray, CT... these medical imaging examinations." The title of paragraph 2 is: "Application of NIR probes in tablet production control." The title of paragraph 3 is: "The most effective medicine for children's cough and asthma."
[0060] like Figure 4 As shown, the total text size of each paragraph is 15k, but the total text size of the paragraph headings is only 1.5k. Therefore, the computational resources and time required for subsequent steps can be significantly reduced.
[0061] Determining the paragraph title for each paragraph can also be achieved using third-party tools such as Python, which will not be detailed here. Furthermore, establishing an index from each paragraph title to the corresponding paragraph can be done by creating key-value pairs. For example, for any given paragraph, use the paragraph title as the key and the paragraph itself as the value to obtain a key-value pair. This allows you to subsequently find the paragraph corresponding to the title using the key.
[0062] Furthermore, in this embodiment, the inference task can be a question-and-answer task, such as a question. The length of the text data can exceed the reference length, that is, the text data is long text data. In other words, the method provided in this embodiment can be applied to scenarios where answers to question-and-answer tasks are obtained through long text data. For example, in long document question-and-answer applications, using the method provided in this embodiment to obtain answers to user-initiated questions can not only avoid wasting computing resources during the answer acquisition process of long document question-and-answer applications, but also shorten the total time of the end-to-end process, preventing users from having to wait a long time to get answers after initiating a question.
[0063] Optionally, the reasoning task in step 301 can also be other types of tasks besides question-and-answer tasks, such as text generation tasks, which will not be illustrated here.
[0064] Step 302: Determine the relevance of each paragraph to the reasoning task based on the paragraph summary of each paragraph and the relevance of the paragraph summary to the reasoning task. The content of the paragraph summary is a portion of the content extracted from each paragraph.
[0065] In other words, for any paragraph among multiple paragraphs, the relevance of that paragraph to the reasoning task is determined based on the relevance of its paragraph summary to the reasoning task. The content of the paragraph summary is a portion of the content extracted from that paragraph.
[0066] Each paragraph's summary is either the paragraph title or a summary of the paragraph, while the paragraph summary is the abridged version of each paragraph. For example, if multiple paragraphs include the first paragraph, then the first paragraph's summary is either the paragraph title or a summary of the first paragraph, and the first paragraph's summary is the abridged version of the first paragraph.
[0067] For any given paragraph, since the text volume of its heading or summary is less than the text volume of the paragraph itself, the computational resources required to determine the relevance of the paragraph summary to the reasoning task are less than those required to determine the relevance of the paragraph itself to the reasoning task. Correspondingly, the time spent determining the relevance of the paragraph summary to the reasoning task is less than the time spent determining the relevance of the paragraph itself to the reasoning task.
[0068] The following explanation uses the first paragraph out of a set of paragraphs as an example to illustrate step 302. The relevance between the paragraph and the inference task can be obtained for the other paragraphs in the text data using the same method, and will not be elaborated upon further.
[0069] In some embodiments, determining the relevance of each paragraph to the reasoning task based on the relevance of each paragraph summary to the reasoning task can be achieved as follows: For the first paragraph among multiple paragraphs, determine whether the relevance of the first paragraph title to the reasoning task is greater than a first threshold; if the relevance of the first paragraph title to the reasoning task exceeds the first threshold, use the relevance of the first paragraph title to the reasoning task as the relevance of the first paragraph to the reasoning task; if the relevance of the first paragraph title to the reasoning task is lower than the first threshold, determine the relevance of the first paragraph summary to the reasoning task, and if the relevance of the first paragraph summary to the reasoning task exceeds a second threshold, use the relevance of the first paragraph summary to the reasoning task as the relevance of the first paragraph to the reasoning task.
[0070] In other words, in this embodiment, when the relevance of the paragraph title of the first paragraph to the reasoning task exceeds a first threshold, the paragraph title of the first paragraph is used as the paragraph summary of the first paragraph. When the relevance of the paragraph title of the first paragraph to the reasoning task is lower than the first threshold, the paragraph summary of the first paragraph is used to determine the paragraph outline of the first paragraph.
[0071] The relevance of the paragraph title of the first paragraph to the inference task, as well as the relevance of the paragraph summary of the first paragraph to the inference task, can both be achieved through a recall model. Examples of recall models include remanker models or vector library retrieval models, which will not be described in detail here.
[0072] Furthermore, considering that the first threshold is used to determine whether the paragraph title of the first paragraph is suitable as a paragraph summary of the first paragraph, and the second threshold is used to determine whether the paragraph summary of the first paragraph is suitable as a paragraph summary of the first paragraph, that is, both the first threshold and the second threshold are used to determine whether a certain piece of information is suitable as a paragraph summary of the first paragraph, based on this, the first threshold and the second threshold can be set to the same threshold. Optionally, in the embodiments of this application, the first threshold and the second threshold can also be set to different thresholds, and the embodiments of this application do not limit this.
[0073] In addition, after determining the relevance of the paragraph summary of the first paragraph to the reasoning task, the first paragraph is discarded if the relevance of the paragraph summary of the first paragraph to the reasoning task is lower than the second threshold.
[0074] If the relevance of the paragraph summary of the first paragraph to the reasoning task is lower than the second threshold, it indicates that the correlation between the paragraph summary of the first paragraph and the reasoning task is relatively weak, so the first paragraph can be discarded.
[0075] Discarding the first paragraph can be understood as: the first paragraph is not considered as a candidate paragraph for the subsequent selection of the target number of paragraphs.
[0076] Additionally, in this embodiment, a third threshold can be set. In this scenario, when the relevance of the title of the first paragraph to the inference task is lower than the first threshold but higher than the third threshold, the relevance of the paragraph summary of the first paragraph to the inference task is determined. Correspondingly, when the relevance of the title of the first paragraph to the inference task is lower than the third threshold, the first paragraph is discarded.
[0077] In this embodiment, when the relevance between the paragraph title of the first paragraph and the inference task is too low, it is possible that all content in the first paragraph has a low relevance to the inference task. In this scenario, it is unnecessary to calculate the relevance between the paragraph summary of the first paragraph and the inference task. Therefore, this embodiment also sets a third threshold. Thus, only when the relevance between the paragraph title of the first paragraph and the inference task falls between the first threshold and the third threshold will the relevance between the paragraph summary of the first paragraph and the inference task be further calculated; otherwise, the first paragraph is discarded.
[0078] In some embodiments, a set of candidate paragraphs can be set up. When the relevance between the paragraph title of the first paragraph and the reasoning task exceeds a first threshold, the first paragraph is added to the set of candidate paragraphs as a candidate paragraph, and the relevance between the paragraph title of the first paragraph and the reasoning task is used as the relevance between the first paragraph and the reasoning task.
[0079] Accordingly, when the relevance between the paragraph title of the first paragraph and the reasoning task falls between the first threshold and the third threshold, the relevance between the paragraph summary of the first paragraph and the reasoning task is determined. When the relevance between the paragraph summary of the first paragraph and the reasoning task exceeds the second threshold, the first paragraph is added to the candidate paragraph set as a candidate paragraph, and the relevance between the paragraph summary of the first paragraph and the reasoning task is taken as the relevance between the first paragraph and the reasoning task.
[0080] Accordingly, if the relevance between the paragraph title of the first paragraph and the reasoning task is lower than the third threshold, or if the relevance between the paragraph summary of the first paragraph and the reasoning task is lower than the second threshold, the first paragraph is discarded, that is, the first paragraph is not added to the candidate paragraph set as a candidate paragraph.
[0081] Optionally, in some other embodiments, a third threshold may not be set. In this case, if the relevance of the paragraph title of the first paragraph to the reasoning task is lower than the first threshold, regardless of the relevance value of the paragraph title of the first paragraph to the reasoning task, the relevance of the paragraph summary of the first paragraph to the reasoning task is directly calculated.
[0082] In addition, in the embodiments of this application, "exceeding" can be understood as greater than or equal to, and "below" can be understood as less than. Optionally, in the embodiments of this application, "exceeding" can be understood as greater than, and "below" can be understood as less than or equal to.
[0083] Additionally, the paragraph summary of the first paragraph is the paragraph after deleting the first paragraph. In some embodiments, determining the paragraph summary of the first paragraph can be achieved by: determining the perplexity (PPL) of each word in the multiple words included in the first paragraph; deleting words with PPL lower than the PPL threshold from the first paragraph to obtain the paragraph summary of the first paragraph.
[0084] The PPL (Programmatical Point of Interest) for each lexical unit indicates the degree of association between that lexical unit and other lexical units in the first paragraph. The PPL threshold is a pre-set threshold. For any lexical unit in the PPL, if its PPL is lower than the threshold, it is considered redundant and can be deleted. Conversely, if its PPL exceeds the threshold, it is considered non-redundant and is retained.
[0085] The PPL (Problem-Pulse Parameter) for each lexical unit can be determined using a PPL model. For example, for any lexical unit in the first paragraph, the lexical unit is input into the PPL model. The PPL model analyzes the relationship between this lexical unit and other lexical units in the first paragraph, and then outputs the PPL for that lexical unit. Optionally, the lexical unit can also be input into the PPL model along with the reasoning task. The PPL model analyzes the relationship between this lexical unit and other lexical units in the first paragraph, as well as the relationship between this lexical unit and the reasoning task, and then outputs the PPL for that lexical unit.
[0086] The embodiments of this application do not provide a detailed description of the structure and reasoning process of the PPL model.
[0087] Optionally, the paragraph summary of the first paragraph can also be determined by: determining the similarity between each word in the multiple words included in the first paragraph and the reasoning task; deleting words with similarity lower than the similarity threshold from the first paragraph to obtain the paragraph summary of the first paragraph.
[0088] In other words, the embodiments of this application can use any method to trim the first paragraph to determine the paragraph summary of the first paragraph. Examples will not be provided here.
[0089] Figure 5 This is a flowchart illustrating a method for trimming the first paragraph, as provided in an embodiment of this application. Figure 5As shown, the title of the first paragraph is "Overview". Since the title of the paragraph has a low relevance to the reasoning task, the content of the first paragraph is truncated to obtain the truncated first paragraph (compressed context). Then, the recall model is used to calculate the relevance between the truncated first paragraph (i.e., paragraph summary) and the reasoning task.
[0090] Furthermore, the above example assumes that the first paragraph has a paragraph title. Optionally, after analyzing the text data, if it is detected that the first paragraph does not have a paragraph title, the first paragraph can be directly cropped to obtain a paragraph summary, and then the relevance between the first paragraph and the reasoning task can be determined based on the paragraph summary.
[0091] Furthermore, the above example uses the paragraph summary of the first paragraph as the paragraph title or paragraph summary of the first paragraph. Optionally, in this embodiment, the paragraph summary of the first paragraph can also be other data that can represent the basic content of the first paragraph, which will not be illustrated here. In other words, in this embodiment, the paragraph summary of the first paragraph can indicate the basic content of the first paragraph, and the size of the paragraph summary of the first paragraph is smaller than the size of the first paragraph itself.
[0092] Step 303: Based on the relevance of each paragraph to the reasoning task, determine the target number of paragraphs from multiple paragraphs.
[0093] Here, the target number can be k, and the purpose of step 303 is to select the top-k paragraphs from the multiple paragraphs. In other words, the selected target number of paragraphs are more relevant to the inference task than the other paragraphs among the multiple paragraphs.
[0094] In some embodiments, step 303 can be implemented by: sorting multiple paragraphs in descending order of relevance to the inference task; and selecting the top-ranked target paragraphs. Optionally, multiple paragraphs can also be sorted in ascending order of relevance to the inference task; and the bottom-ranked target paragraphs can be selected.
[0095] In addition, in the scenario where a set of candidate paragraphs is determined in step 302, the target number of paragraphs can be determined from the set of candidate paragraphs based on the relevance of each candidate paragraph to the inference task, thereby further saving computing resources.
[0096] Step 304: Input the target number of paragraphs and the inference task into the LLM so that the LLM can output the inference results of the inference task based on the target number of paragraphs.
[0097] In some embodiments, a target number of paragraphs and the inference task can be combined into a cue word, and the cue word can be input into an LLM so that the LLM outputs the inference result of the inference task based on the cue word.
[0098] The following is based on Figure 6 The preceding steps 301 to 304 will be illustrated using an example. It should be noted that... Figure 6 This is an example of the aforementioned steps 301 to 304, and does not constitute a limitation on the aforementioned steps 301 to 304.
[0099] like Figure 6 As shown, the method for addressing this problem in LLM provided in this application includes the following steps.
[0100] Step 6.1: Input the question to be reasoned (i.e., the question-answering task) and text data.
[0101] Step 6.2: Parse the text data to obtain multiple paragraphs. For any paragraph, determine whether the paragraph title can be parsed. If the paragraph title can be parsed, proceed to step 6.3. If the paragraph title cannot be parsed, proceed to step 6.8.
[0102] Step 6.3: If the paragraph title can be parsed, extract the paragraph title of each paragraph and create an index from paragraph title to paragraph. Calculate the relevance between the paragraph title and the question.
[0103] Step 6.4: Determine whether the relevance between the paragraph title and the question is greater than γ (i.e., the first threshold).
[0104] Step 6.5: If the relevance between the paragraph title and the question is greater than γ, find the paragraph corresponding to the paragraph title, save the relevance as the relevance between the paragraph and the question, and add the paragraph to the candidate paragraph set.
[0105] Step 6.6: If the relevance between the paragraph title and the question is less than or equal to γ, further determine whether the relevance between the paragraph title and the question is less than θ (i.e., the third threshold).
[0106] Step 6.7: If the relevance between the paragraph title and the question is less than θ, then discard the paragraph.
[0107] Step 6.8: If the relevance between the paragraph title and the question is greater than or equal to θ, then prune the paragraph to obtain the pruned paragraph (i.e., the paragraph summary). For example, redundant words in the paragraph can be pruned based on the PPL of each word in the paragraph.
[0108] Step 6.9: Calculate the relevance between the trimmed paragraph and the question, that is, calculate the relevance between the paragraph summary and the question.
[0109] Step 6.10: Determine whether the relevance between the paragraph summary and the question is greater than γ, where γ is the aforementioned second threshold.
[0110] Step 6.11: If the relevance between the paragraph summary and the question is greater than γ, find the paragraph corresponding to the paragraph summary, save the relevance as the relevance between the paragraph and the question, and add the paragraph to the candidate paragraph set.
[0111] Step 6.12: If the relevance of the paragraph summary to the question is less than or equal to γ, then discard the paragraph.
[0112] Step 6.13: Sort the paragraphs in the candidate paragraph set according to their relevance to the question, and then select the top-k paragraphs, that is, select the target number of paragraphs.
[0113] Step 6.14: Reorganize the target number of paragraphs and the question into prompt words, input the prompt words into the LLM, and have the LLM output the reasoning result for the question.
[0114] In summary, in this embodiment, the relevance of each paragraph to the inference task is determined by utilizing the paragraph summary of each paragraph and its relevance to the inference task, thereby identifying a target number of paragraphs from multiple paragraphs. Since the paragraph summary is a partial extraction from each paragraph, it is unnecessary to calculate the relevance of the entire content of each paragraph to the inference task. This saves computational resources, reducing memory usage and improving throughput; it also shortens the time required to filter the target number of paragraphs from text data, thereby reducing the overall end-to-end process time and preventing users from having to wait a long time for an answer after initiating a question-and-answer task.
[0115] In addition, this application also provides a schematic diagram of the structure of an execution device for an LLM inference task, as shown in the embodiment. Figure 7 As shown, the device 700 includes the following modules.
[0116] Module 701 is used to acquire the inference task and text data, which includes multiple paragraphs; for specific implementation details, please refer to [link / reference]. Figure 3 Step 301 in the embodiment.
[0117] The first determining module 702 is used to determine the relevance of each paragraph to the inference task based on the relevance of the paragraph summary to the inference task. The content of the paragraph summary is a portion extracted from each paragraph. For specific implementation details, please refer to [reference needed]. Figure 3 Step 302 in the embodiment.
[0118] The second determination module 703 is used to determine the target number of paragraphs from multiple paragraphs based on the relevance of each paragraph to the inference task; for specific implementation details, please refer to [reference needed]. Figure 3 Step 303 in the embodiment.
[0119] The inference module 704 is used to input the target number of paragraphs and the inference task into the LLM, so that the LLM can output the inference results of the inference task based on the target number of paragraphs. For specific implementation details, please refer to [reference needed]. Figure 3 Step 304 in the embodiment.
[0120] In some embodiments, the paragraph summary for each paragraph is the paragraph title or paragraph summary for each paragraph, which is the paragraph after each paragraph has been shortened.
[0121] In some embodiments, the first determining module is configured to:
[0122] For the first paragraph among multiple paragraphs, determine whether the relevance of the paragraph title of the first paragraph to the reasoning task is greater than a first threshold;
[0123] When the relevance of the paragraph title of the first paragraph to the reasoning task exceeds the first threshold, the relevance of the paragraph title of the first paragraph to the reasoning task is taken as the relevance of the first paragraph to the reasoning task.
[0124] When the relevance of the paragraph title of the first paragraph to the reasoning task is lower than the first threshold, the relevance of the paragraph summary of the first paragraph to the reasoning task is determined. When the relevance of the paragraph summary of the first paragraph to the reasoning task exceeds the second threshold, the relevance of the paragraph summary of the first paragraph to the reasoning task is taken as the relevance of the first paragraph to the reasoning task.
[0125] In some embodiments, the device further includes:
[0126] The first discard module is used to discard the first paragraph when the relevance of the paragraph summary to the reasoning task in the first paragraph is lower than a second threshold.
[0127] In some embodiments, the first determining module is configured to:
[0128] The relevance of the paragraph summary of the first paragraph to the reasoning task is determined when the relevance of the paragraph title of the first paragraph to the reasoning task is lower than the first threshold but higher than the third threshold.
[0129] In some embodiments, the device further includes:
[0130] The second discard module is used to discard the first paragraph when the relevance of the paragraph title of the first paragraph to the reasoning task is lower than a third threshold.
[0131] In some embodiments, the device further includes:
[0132] The third determination module is used to determine the perplexity PPL of each word in the first paragraph among multiple paragraphs.
[0133] The trimming module is used to remove words with a PPL less than the PPL threshold from the first paragraph, thus obtaining a paragraph summary of the first paragraph.
[0134] In some embodiments, the reasoning task is a question-and-answer task, and the length of the text data exceeds the reference length.
[0135] In summary, in this embodiment, the relevance of each paragraph to the inference task is determined by utilizing the paragraph summary of each paragraph and its relevance to the inference task, thereby identifying a target number of paragraphs from multiple paragraphs. Since the paragraph summary is a partial extraction from each paragraph, it is unnecessary to calculate the relevance of the entire content of each paragraph to the inference task. This saves computational resources, reducing memory usage and improving throughput; it also shortens the time required to filter the target number of paragraphs from text data, thereby reducing the overall end-to-end process time and preventing users from having to wait a long time for an answer after initiating a question-and-answer task.
[0136] It should be noted that the LLM inference task execution device provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the LLM inference task execution device and the LLM inference task execution method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.
[0137] In addition, embodiments of this application also provide a computing device cluster, including at least one computing device, each computing device including a processor and a memory;
[0138] The processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the cluster of computing devices executes the above-described method for executing the inference task of the LLM.
[0139] Figure 8This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Figure 8 As shown, the computing device includes at least one processor 801, a communication bus 802, a memory 803, and at least one communication interface 804.
[0140] The processor 801 may be a general-purpose central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present application.
[0141] The communication bus 802 may include a path for transmitting information between the aforementioned components.
[0142] The memory 803 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or it may be an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital versatile optical discs, Blu-ray discs, etc.), a magnetic disk or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. The memory 803 may exist independently and be connected to the processor 801 via a communication bus 802. The memory 803 may also be integrated with the processor 801.
[0143] The memory 803 stores program code for executing the scheme of this application, and its execution is controlled by the processor 801. The processor 801 executes the program code stored in the memory 803. The program code may include one or more software modules. The foregoing embodiments can use the processor 801 and one or more software modules in the program code in the memory 803 to determine the data used for application development.
[0144] The communication interface 804 uses any transceiver-like device to communicate with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc.
[0145] In a specific implementation, as one example, a computing device may include multiple processors, for example... Figure 8 The processors 801 and 805 are shown. Each of these processors can be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. Here, "processor" can refer to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).
[0146] In a specific implementation, as one embodiment, the computing device may further include an output device 806 and an input device 807. The output device 806 communicates with the processor 801 and can display information in various ways. For example, the output device 806 may be a liquid crystal display (LCD), a light-emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector, etc. The input device 807 communicates with the processor 801 and can receive user input in various ways. For example, the input device 807 may be a mouse, keyboard, touchscreen device, or sensing device, etc.
[0147] The aforementioned computing device can be a general-purpose computing device or a dedicated computing device. In specific implementations, the computing device can be a desktop computer, a portable computer, a network server, a handheld digital assistant (PDA), a mobile phone, a tablet computer, a wireless terminal device, a communication device, or an embedded device. This application does not limit the type of computing device.
[0148] In addition, this application also provides a computer program product containing instructions that, when the instructions are run by a computing device cluster, cause the computing device cluster to execute the above-described method for executing the LLM inference task.
[0149] In addition, embodiments of this application also provide a computer-readable storage medium including computer program instructions, which, when executed by a computing device cluster, enable the computing device cluster to execute the above-described method for executing the inference task of an LLM.
[0150] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital versatile discs (DVDs)), or semiconductor media (e.g., solid-state disks (SSDs)).
[0151] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.
[0152] The terms "first" and "second," etc., used in the specification and claims of this application are used to distinguish different objects, not to describe a specific order of objects, and should not be construed as indicating or implying relative importance.
[0153] In the description of the embodiments in this application, unless otherwise stated, "at least one" means one or more. "More than one" means two or more.
[0154] A references B, which means that A is the same as B or A is a simple variation of B.
[0155] In this application, the term "and / or" is merely a description of the relationship between related objects, indicating that there are three possible relationships. For example, A and / or B means: A exists alone, A and B exist simultaneously, or B exists alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.
[0156] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.
Claims
1. A method for executing inference tasks in a large language model (LLM), characterized in that, The method includes: Acquire inference tasks and text data, wherein the text data includes multiple paragraphs; The relevance of each paragraph to the reasoning task is determined based on the paragraph summary of each paragraph and the relevance of the paragraph summary to the reasoning task. The content of the paragraph summary is a portion of the content extracted from each paragraph. Based on the relevance of each paragraph to the reasoning task, a target number of paragraphs are determined from the plurality of paragraphs; The target number of paragraphs and the inference task are input into the LLM, so that the LLM outputs the inference result of the inference task based on the target number of paragraphs.
2. The method as described in claim 1, characterized in that, The paragraph summary of each paragraph is the paragraph title or paragraph summary of each paragraph, and the paragraph summary is the paragraph after each paragraph has been shortened.
3. The method as described in claim 1 or 2, characterized in that, The step of determining the relevance of each paragraph to the inference task based on the paragraph summary of each paragraph includes: For the first paragraph among the plurality of paragraphs, determine whether the relevance of the paragraph title of the first paragraph to the reasoning task is greater than a first threshold; When the relevance between the paragraph title of the first paragraph and the reasoning task exceeds a first threshold, the relevance between the paragraph title of the first paragraph and the reasoning task is taken as the relevance between the first paragraph and the reasoning task. When the relevance of the paragraph title of the first paragraph to the reasoning task is lower than the first threshold, the relevance of the paragraph summary of the first paragraph to the reasoning task is determined. When the relevance of the paragraph summary of the first paragraph to the reasoning task exceeds the second threshold, the relevance of the paragraph summary of the first paragraph to the reasoning task is taken as the relevance of the first paragraph to the reasoning task.
4. The method as described in claim 3, characterized in that, After determining the relevance of the paragraph summary of the first paragraph to the reasoning task, the method further includes: If the relevance of the paragraph summary of the first paragraph to the reasoning task is lower than the second threshold, the first paragraph is discarded.
5. The method as described in claim 3 or 4, characterized in that, The step of determining the relevance of the paragraph summary of the first paragraph to the reasoning task when the relevance of the paragraph title of the first paragraph to the reasoning task is lower than the first threshold includes: When the relevance of the paragraph title of the first paragraph to the reasoning task is lower than the first threshold but higher than the third threshold, the relevance of the paragraph summary of the first paragraph to the reasoning task is determined.
6. The method as described in claim 5, characterized in that, The method further includes: The first paragraph is discarded when the relevance of its title to the reasoning task is lower than the third threshold.
7. An execution device for reasoning tasks in a Large Language Model (LLM), characterized in that, The device includes: The acquisition module is used to acquire inference tasks and text data, wherein the text data includes multiple paragraphs; The first determining module is used to determine the relevance of each paragraph to the reasoning task based on the relevance of the paragraph summary of each paragraph to the reasoning task, wherein the content of the paragraph summary is a portion of the content extracted from each paragraph; The second determining module is used to determine a target number of paragraphs from the plurality of paragraphs based on the relevance of each paragraph to the reasoning task. The inference module is used to input the target number of paragraphs and the inference task into the LLM, so that the LLM outputs the inference result of the inference task based on the target number of paragraphs.
8. The apparatus as claimed in claim 7, characterized in that, The paragraph summary of each paragraph is the paragraph title or paragraph summary of each paragraph, and the paragraph summary is the paragraph after each paragraph has been shortened.
9. The apparatus as claimed in claim 7 or 8, characterized in that, The first determining module is used for: For the first paragraph among the plurality of paragraphs, determine whether the relevance of the paragraph title of the first paragraph to the reasoning task is greater than a first threshold; When the relevance between the paragraph title of the first paragraph and the reasoning task exceeds a first threshold, the relevance between the paragraph title of the first paragraph and the reasoning task is taken as the relevance between the first paragraph and the reasoning task. When the relevance of the paragraph title of the first paragraph to the reasoning task is lower than the first threshold, the relevance of the paragraph summary of the first paragraph to the reasoning task is determined. When the relevance of the paragraph summary of the first paragraph to the reasoning task exceeds the second threshold, the relevance of the paragraph summary of the first paragraph to the reasoning task is taken as the relevance of the first paragraph to the reasoning task.
10. The apparatus as claimed in claim 9, characterized in that, The device further includes: The first discarding module is used to discard the first paragraph when the relevance of the paragraph summary of the first paragraph to the reasoning task is lower than the second threshold.
11. The apparatus as claimed in claim 9 or 10, characterized in that, The first determining module is used for: When the relevance of the paragraph title of the first paragraph to the reasoning task is lower than the first threshold but higher than the third threshold, the relevance of the paragraph summary of the first paragraph to the reasoning task is determined.
12. The apparatus as claimed in claim 11, characterized in that, The device further includes: The second discarding module is used to discard the first paragraph when the relevance of the paragraph title of the first paragraph to the reasoning task is lower than the third threshold.
13. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1-6.
14. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster causes the computing device cluster to perform the method as described in any one of claims 1-6.
15. A computer-readable storage medium, characterized in that, Includes computer program instructions, which, when executed by a cluster of computing devices, perform the method as described in any one of claims 1-6.