Large language model reply content processing method, related hardware and program product

Abstract

The invention provides a large language model reply content processing method, related hardware and a program product. The method comprises the steps that after a large language model obtains initial reply content corresponding to a target question through query according to a first cue word, a second cue word is constructed, and the second cue word is used for guiding the large language model to execute the steps that an extended knowledge text related to the target question is determined; and supplementing the first prompt word based on the initial reply content and the extended knowledge text. And inputting the first cue word and the second cue word into the large language model to enable the large language model to supplement the first cue word according to the second cue word, and querying target reply content corresponding to the target question based on the supplemented first cue word.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a method for processing reply content of a large language model, and related hardware and program products. Background Art

[0002] With the development of artificial intelligence, large language models (LLMs) for text question answering have been widely used. Prompts are an important configuration of large language models, which are used to guide large language models to query content. Obviously, well-designed prompts can make the dialogue in human-computer interaction scenarios more targeted, and the output of large language models more natural and more in line with user needs.

[0003] Therefore, how to optimize the prompt words to improve the quality of the response content of the large language model is the technical problem to be solved by this application. Summary of the invention

[0004] The purpose of this application is to provide a method for processing large language model reply content, related hardware and program products, which can intelligently supplement and optimize the prompt words used in the initial reply content of the query through the large language model, and then re-complete the query based on the supplemented and optimized prompt words to improve the quality of the reply content.

[0005] In order to achieve the above purpose, the embodiment of the present application is implemented as follows: In a first aspect, a method for processing reply content of a large language model is provided, comprising: After the large language model obtains the initial response content corresponding to the target question according to the first prompt word query, a second prompt word is constructed, and the second prompt word is used to guide the large language model to perform: determining an extended knowledge text related to the target question; supplementing the first prompt word based on the initial response content and the extended knowledge text; The first prompt word and the second prompt word are input into the large language model, so that the large language model supplements the first prompt word according to the second prompt word, and obtains the target reply content corresponding to the target question based on the supplemented first prompt word.

[0006] In a second aspect, a device for processing large language model reply content is provided, comprising: A prompt word adaptation module is used to construct a second prompt word after the large language model obtains the initial response content corresponding to the target question according to the first prompt word query, and the second prompt word is used to guide the large language model to execute: determining the extended knowledge text related to the target question; supplementing the first prompt word based on the initial response content and the extended knowledge text; The query supplement module is used to input the first prompt word and the second prompt word into the large language model so that the large language model supplements the first prompt word according to the second prompt word, and obtains the target reply content corresponding to the target question based on the supplemented first prompt word.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor; and a memory configured to store computer-executable instructions, wherein the computer-executable instructions, when executed, cause the processor to execute the method described in the first aspect.

[0008] According to a fourth aspect, a computer program product is provided, the computer program product comprising a computer-readable storage medium storing a computer program, wherein the computer program is operable to enable a computer to implement the method described in the first aspect when executed.

[0009] The embodiment of the present application proposes a solution for processing the reply content of a large language model. After the user completes the initial query through the prompt word, the large language model can be used to intelligently supplement and optimize the prompt word according to the initial reply content queried, so as to query again based on the supplemented and optimized prompt word to improve the quality of the final reply content. Since the process of supplementing and optimizing the prompt word to re-query is completed on the deployment side of the large language model, from the user's perception, it is equivalent to executing only one query, thereby increasing the probability that the first query can hit the user's expectations. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0011] Figure 1 This is a first flow chart of the method for processing large language model reply content according to an embodiment of the present application.

[0012] Figure 2 A schematic diagram of inputting the first prompt word and the second prompt word into a large language model.

[0013] Figure 3 A second flow chart of the method for processing large language model reply content according to an embodiment of the present application.

[0014] Figure 4 This is a third flow chart of the method for processing large language model reply content according to an embodiment of the present application.

[0015] Figure 5 Schematic diagram of the large language model providing feedback on the target question.

[0016] Figure 6 A schematic diagram of the structure of a device for processing large language model reply content according to an embodiment of the present application.

[0017] Figure 7 A schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0018] As mentioned earlier, prompt words are an important configuration of the large language model, which is used to guide the large language model to query content. For example, a user asks the question "How many base stations are there in Beijing?", but by adding the prompt word "the base station must have 5G function", the large language model will more specifically provide the number of 5G base stations deployed in Beijing, rather than providing all base stations deployed in Beijing. Obviously, carefully designed prompt words can make the query results of the large language model more in line with user expectations.

[0019] In actual applications, users often cannot get the expected results in one go by setting prompt words to guide large language model queries, and need to repeatedly adjust the prompt words for multiple queries. In view of this, the present application aims to propose a solution for processing large language model reply content, which can use the large language model to intelligently supplement and optimize the prompt words according to the initial reply content after the user completes the initial query through the prompt words, so as to query again based on the supplemented and optimized prompt words to improve the quality of the final reply content. Since the process of supplementing and optimizing the prompt words to re-query is completed at the deployment end of the large language model, it is equivalent to executing only one query for user perception, thereby increasing the probability that the first query hits the user's expectations.

[0020] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0021] An embodiment of the present application provides a method for processing reply content of a large language model. Figure 1 Schematic diagram of the process, including the following steps: S102, after the large language model obtains the initial response content corresponding to the target question according to the first prompt word query, a second prompt word is constructed, and the second prompt word is used to guide the large language model to execute: determine the extended knowledge text related to the target question; supplement the first prompt word based on the initial response content and the extended knowledge text.

[0022] In this embodiment, the target question is raised by the user, the user sets the first prompt word according to the target question, and obtains the corresponding initial response content by searching according to the first prompt word through the large language model. It should be noted that due to human factors (such as poor consideration), the requirements described in the target question may be defective, resulting in the initial response content searched according to the first prompt word not necessarily meeting the user's expectations.

[0023] To this end, it is necessary to construct a second prompt word for supplementing and optimizing the first prompt word.

[0024] Specifically, this embodiment performs segmentation processing on the plug-in knowledge documents to refine the knowledge texts of different knowledge, and stores these knowledge texts and corresponding keywords in a designated database. In the process of determining the extended knowledge text related to the target question, the second prompt word can guide the large language model to first determine the keywords that match the target question. For example, the large language model can identify the intent of the target question based on natural language capabilities, obtain the question intent corresponding to the target question, and perform keyword extraction on the question intent to obtain the target keywords related to the target question. Afterwards, the second prompt word further guides the large language model to query the designated database for the knowledge text related to each target keyword through keyword matching, so as to supplement the first prompt word as an extended knowledge text.

[0025] In addition, on the basis of the above, redundant judgment of the extended knowledge text can also be added to the second prompt word to avoid duplication of the extended knowledge text with the initial reply content. To achieve redundant judgment, the designated database of this embodiment also stores the fingerprint feature of each knowledge text, which can be, but not limited to, encrypted and calculated based on the hash algorithm (such as the information digest algorithm) to the knowledge text, as the KEY of the knowledge text. Correspondingly, the second prompt word of this embodiment is an appendix to the first prompt word, and the appendix includes a supplementary item set for each target associated word, and the supplementary item includes the extended knowledge text of the target associated word and the fingerprint feature of the extended knowledge text, for example: <knowledge text> + (KEY identifier: 18332F297A57A5A743894A0E4A801FC3). Under the guidance of the second prompt word, the large language model compares the fingerprint feature of the initial reply content with the fingerprint feature of the extended knowledge text associated with each target keyword for similarity to determine whether there is redundant extended knowledge text that repeats the initial reply content. If it is determined that there is redundant extended knowledge text, the supplementary item corresponding to the redundant extended knowledge text in the appendix is ​​removed, and the appendix after removal is added to the first prompt word; if it is determined that there is no redundant extended knowledge text, the appendix is ​​directly added to the first prompt word. It should be noted that in the process of adding key information to the specified database, some basic verification logic can be further set to avoid duplication of the KEY identifier and the information in the knowledge text itself, such as duplication of the replaceable KEY identifier.

[0026] In addition, in actual applications, the second prompt word can also be used to introduce a judgment on whether the first prompt word needs to be supplemented. For example, a preset response content requirement corresponding to the initial response content configuration may include: the number of knowledge blocks of the first prompt word for the initial response content reaches a preset block number threshold, and / or the number of words in the initial response content reaches a preset word number threshold. Correspondingly, under the guidance of the second prompt word, the large language model can first determine whether the initial response content meets the preset response content requirements. If it is determined that the initial response content does not meet the preset response content requirements (such as the number of words in the initial response content is too small, the initial response content has too few knowledge blocks), the above-mentioned step of determining the extended knowledge text related to the target question and the above-mentioned step of supplementing the first prompt word based on the initial response content and the extended knowledge text are performed.

[0027] S104: input the first prompt word and the second prompt word into the large language model, so that the large language model supplements the first prompt word according to the second prompt word, and searches for target reply content corresponding to the target question based on the supplemented first prompt word.

[0028] In this step, the input of the large language model can refer to Figure 2shown.

[0029] In addition to the target question, the first prompt word may also include some external professional knowledge documents required for querying the target question. The external professional knowledge documents may come from the above-mentioned designated database and be represented in the form of fingerprint features (key: XXX).

[0030] It should be noted that linking a prompt word with a professional knowledge document is a normal function of a prompt word, but the extended knowledge document linked to the second prompt word in this embodiment is to supplement the first prompt word, while the professional knowledge document linked to the first prompt word is to answer the target question, and the two have different functions.

[0031] Specifically, the second prompt word mainly includes the following elements: Element 1: Adding the conditional setting of appendix; prompting the large language model to ignore the knowledge documents of the first prompt word when generating the reply content to avoid duplication. In addition, according to the main content of the large language model's reply, the large language model can be instructed to add an appendix if the initial reply content does not meet the preset reply content requirements.

[0032] Element 2: The first paragraph of the appendix is ​​fixed as "The following information is for reference". If the appendix of the first prompt word cannot be generated later (such as finding any target keyword or finding no extended knowledge document associated with any target keyword), the appendix is ​​deleted and the original initial response content is directly used as the query result of the target question.

[0033] Element 3: Add the identified target keywords to the appendix; and instruct the large language model to list each target keyword in a separate line, using the supplementary item format of "<target keyword:> (knowledge text associated with the target keyword)".

[0034] Element 4: The appendix lists the fingerprint features of all extended knowledge texts. This fingerprint feature list is intended to avoid duplication between the extended knowledge text and the initial response content, and provides deduplication by comparing the fingerprint feature similarity.

[0035] Based on the guidance of the first prompt word and the second prompt word (the appendix of the first prompt word), the large language model can perform Figure 3 The process shown: S1, judging whether the first prompt word needs to be supplemented according to whether the initial reply content meets the preset reply content requirements.

[0036] As mentioned above, the preset reply content requirements may be but are not limited to: the number of knowledge blocks of the first prompt word for the initial reply content reaches a preset block number threshold, and / or the number of words in the initial reply content reaches a preset word number threshold.

[0037] Here, assuming that the first prompt word needs to be supplemented, the subsequent steps are as follows: S2, determine the target keywords related to the target question.

[0038] This step performs intent recognition on the target question to obtain the question intent corresponding to the target question, and performs keyword extraction on the question intent to obtain target keywords related to the target question.

[0039] S3, querying the designated database for the extended knowledge text matching the target keyword and the fingerprint feature corresponding to the extended knowledge text, so as to create a supplementary item and fingerprint feature list for each target keyword in the appendix.

[0040] The specific format of the appendix is ​​as follows: Figure 2 As shown, no further description is given here.

[0041] S4, determining the fingerprint features of the initial reply content, and performing a similarity comparison between the fingerprint features of the initial reply content and the fingerprint features of each extended knowledge text recorded in the fingerprint feature list, so as to filter out the supplementary items of the redundant extended knowledge text in the appendix.

[0042] It should be noted that, in this embodiment, the same encoding method as the fingerprint feature of the extended knowledge text (such as the information digest algorithm) can be used to perform encryption calculation on the initial reply content to obtain the fingerprint feature of the initial reply content.

[0043] In addition, in the case where the first prompt word includes the vector features of the professional knowledge text; if the professional knowledge text comes from a specified database, the fingerprint features of the initial reply content may also include the vector features of the professional knowledge text; if the professional knowledge text comes from other databases, considering that the encoding method of the vector features in other databases is different from the encoding method of the fingerprint features in the specified database, it is necessary to first obtain the professional knowledge text from other databases based on the vector features of the professional knowledge text, and use the same encoding method as the fingerprint features of the extended knowledge text to encrypt the obtained professional knowledge text to obtain the fingerprint features of the professional knowledge text, and the fingerprint features of the professional knowledge text are also regarded as the fingerprint features of the initial reply content.

[0044] S5, conduct a quality score for the extended knowledge text of each supplementary item in the appendix, and remove supplementary items in the appendix whose quality score does not meet the preset score requirements.

[0045] The method of performing quality scoring on the extended knowledge text in this step may be, but is not limited to, at least one of the following: 1) Based on the similarity between the semantics of the extended knowledge text of each supplementary item in the appendix and the semantics of the target question, the quality of the extended knowledge text of each supplementary item in the appendix is ​​scored; and / or, wherein the higher the similarity, the higher the score, and the lower the similarity, the lower the score; since calculating the similarity between semantics is a common function of a large language model, it will not be repeated in this article.

[0046] 2) Based on the correlation between the field to which the extended knowledge text of each supplementary item in the appendix belongs and the field to which the target question belongs, the quality of the extended knowledge text of each supplementary item in the appendix is ​​scored; the correlation between fields is the preset prior knowledge, the higher the correlation, the higher the score, and the lower the similarity, the lower the score.

[0047] S6, delete the fingerprint feature list (jason array) in the appendix and the fingerprint features in the supplementary items to use the remaining content in the appendix as part of the target reply content.

[0048] It should be noted that this step only deletes the fingerprint feature list and the fingerprint features in the jason array and supplementary items in the initial appendix, rather than guiding the large language model to focus on querying new reply content. Therefore, it is not easily restricted by the tokens output of the large language model data, further reducing the dependence on the output quality of the large language model. In addition, the remaining content in the appendix is ​​all natural language description content, which is suitable for users to view directly.

[0049] The specific application of the method of this embodiment is introduced below.

[0050] The method of this embodiment can be applied to the communication field. For specific application scenarios, refer to Figure 1 As shown, the communication user asks the communication operator's human-computer interaction robot a target question related to the communication service, and the human-computer interaction robot uses a large language model to query the corresponding reply content and feed back to the communication user.

[0051] First, refer to Figure 4As shown in the figure, the solid arrow represents the traditional human-computer interaction process. When the human-computer interaction machine receives the target question raised by the communication user, it uses the retrieval-augmented generation (RAG) technology to connect the professional knowledge text in the communication field to construct the first prompt word in combination with the target question. Among them, these professional knowledge texts are stored in the professional knowledge database, and the professional knowledge database records each professional knowledge text and the corresponding vector feature. The professional knowledge text connected to the first prompt word is associated with the vector feature. For the large language model, the professional knowledge text stored in the professional knowledge database can be found based on the vector feature in the first prompt word. The purpose of introducing the professional knowledge text in RAG is to enable the large language model to understand and directly obtain the professional knowledge in the communication field without the need for fine-tuning training, thereby providing guidance for answering the target question. Afterwards, the human-computer interaction machine inputs the first prompt word (including the target question and the connected professional knowledge text) into the large language model to query and obtain the initial reply content.

[0052] It should be understood that the target question specifically includes some of the communication user's needs. Due to human factors, the target question may not be described in a comprehensive manner, which may result in the initial reply content not meeting the communication user's expectations. To this end, it is necessary to combine the method of this embodiment to supplement and optimize the first prompt word. The supplementary optimization process is Figure 4 The specific process of the post-prompt word optimization module shown is represented by the dotted arrow: first, the large language model constructs a second prompt word specifically used to supplement the first prompt word based on the first prompt word and the initial response content; then, the second prompt word and the first prompt word are input into the large language model after being processed by RAG to try to query the response content of the target question again.

[0053] It should be understood that the large language model supplements and optimizes the first prompt word under the guidance of the second prompt word, and then tries to query again based on the supplemented and optimized first prompt word, and feeds back the target reply content obtained by the second query to the communication user.

[0054] In addition, based on the foregoing, it can be known that the large language model, under the guidance of the second prompt word, needs to find the extended knowledge text from the designated database. The designated database may be, but is not limited to, the professional knowledge database used by RAG. If the designated database and the professional knowledge database are the same database, then the fingerprint features of the extended knowledge text and described in this article are the vector features of the professional knowledge text text. Under the guidance of the second prompt word of the large language model, the vector features of the professional knowledge text attached to RAG can be determined as at least part of the fingerprint features corresponding to the initial reply content. If the designated database and the professional knowledge database are not the same database, then under the guidance of the second prompt word of the large language model, it is necessary to use the encoding method of the fingerprint features of the extended knowledge text and to encode the initial reply content to obtain the fingerprint features in the initial reply.

[0055] Reference here Figure 5 As shown in the figure, it is assumed that the target question raised by the communication user is: “How much does package A for XX service cost?”

[0056] The responses obtained based on the traditional query method are as follows: "XXX Package A can be stacked multiple times and is valid for one year. Points are awarded based on successful sending and reach, and unsuccessful text messages are free of charge. Successfully delivered messages are charged at 0.05 yuan per message, and unanswered calls are charged according to message reach."

[0057] The method of this embodiment is to supplement the appendix content on the basis of the initial reply content. Therefore, when the reply content obtained by the traditional query is used as the initial reply content, the target reply content queried after the prompt words are supplemented and optimized is additionally obtained. Figure 5 The appendix shown expands the information of XXX Package B to facilitate users to compare between Package A and Package B.

[0058] In addition, corresponding to Figure 1 The method shown, another embodiment of this embodiment also provides a processing device for large language model reply content. Figure 6 is a schematic diagram of the structure of the processing device 600, comprising: The prompt word adaptation module 610 is used to construct a second prompt word after the large language model obtains the initial response content corresponding to the target question according to the first prompt word query, and the second prompt word is used to guide the large language model to execute: determine the extended knowledge text related to the target question; supplement the first prompt word based on the initial response content and the extended knowledge text.

[0059] The query supplement module 620 is used to input the first prompt word and the second prompt word into the large language model so that the large language model supplements the first prompt word according to the second prompt word, and obtains the target response content corresponding to the target question based on the supplemented first prompt word.

[0060] After the user completes the initial query through the prompt word, the device of this embodiment can use the large language model to intelligently supplement and optimize the prompt word according to the initial reply content found in the query, so as to query again based on the supplemented and optimized prompt word to improve the quality of the final reply content. Since the process of supplementing and optimizing the prompt word to re-query is completed on the deployment side of the large language model, from the user's perception, it is equivalent to executing only one query, thereby increasing the probability that the first query can hit the user's expectations.

[0061] Optionally, the second prompt word is also used to guide the large language model to determine whether the initial response content meets the preset response content requirements, and if the initial response content does not meet the preset response content requirements, determine the extended knowledge text related to the target question, and supplement the first prompt word based on the initial response content and the extended knowledge text.

[0062] Optionally, the preset reply content requirement includes that the number of knowledge blocks of the first prompt word for the initial reply content reaches a preset block number threshold, and / or the number of words in the initial reply content reaches a preset word number threshold.

[0063] Optionally, the large language model determines the extended knowledge text related to the target question, including: determining at least one target keyword related to the target question; querying a designated database for each knowledge text related to the target keyword as the extended knowledge text; wherein the designated database stores multiple knowledge texts and the correspondence between the multiple knowledge texts and the keywords.

[0064] Optionally, the large language model determines at least one target keyword related to the target question, including: performing intent recognition on the target question to obtain a question intent corresponding to the target question; performing keyword extraction on the question intent to obtain a target keyword related to the target question.

[0065] Optionally, the designated database also stores fingerprint features of each knowledge text; the second prompt word serves as an appendix to the first prompt word, the appendix includes supplementary items set for each of the target associated words, the supplementary items include the extended knowledge text of the target associated word and the fingerprint features of the extended knowledge text; the large language model supplements the first prompt word based on the initial reply content and the extended knowledge text, including: comparing the fingerprint features of the initial reply content with the fingerprint features of the extended knowledge text associated with each of the target keywords for similarity to determine whether there is redundant extended knowledge text that is repeated with the initial reply content; if it is determined that there is redundant extended knowledge text, then removing the supplementary items corresponding to the redundant extended knowledge text in the appendix, so as to supplement the appendix after removal to the first prompt word; if it is determined that there is no redundant extended knowledge text, then directly supplementing the appendix to the first prompt word.

[0066] Optionally, the second prompt word is also used to guide the large language model to execute: when an appendix to the first prompt word cannot be generated, directly using the initial response content as a query result of the target question.

[0067] Optionally, the second prompt word is also used to guide the large language model to execute: when there is no supplementary item in the appendix, directly use the initial response content as the query result of the target question.

[0068] Optionally, the appendix also includes a fingerprint feature list, which includes the fingerprint features of the extended knowledge text associated with each of the target associated words; the large language model compares the fingerprint features of the initial reply content with the fingerprint features of the extended knowledge text associated with each of the target keywords for similarity to determine whether there is redundant extended knowledge text that is repeated with the initial reply content, including: comparing the fingerprint features of the initial reply content with the fingerprint features recorded in the fingerprint feature list for similarity to determine whether there is redundant extended knowledge text that is repeated with the initial reply content.

[0069] Optionally, the second prompt word is also used to guide the large language model to execute: deleting the fingerprint feature list in the appendix and the fingerprint features in the supplementary items, so as to use the remaining content in the appendix as part of the target reply content.

[0070] Optionally, the second prompt word is also used to guide the large language model to execute: before deleting the fingerprint feature list in the appendix and the fingerprint features in the supplementary items, quality scoring the extended knowledge text of each supplementary item in the appendix, and eliminating supplementary items in the appendix whose quality scores do not meet preset scoring requirements.

[0071] Optionally, the large language model performs quality scoring on the extended knowledge text of each supplementary item in the appendix, including: performing quality scoring on the extended knowledge text of each supplementary item in the appendix based on the similarity between the semantics of the extended knowledge text of each supplementary item in the appendix and the semantics of the target question; and / or performing quality scoring on the extended knowledge text of each supplementary item in the appendix based on the correlation between the field to which the extended knowledge text of each supplementary item in the appendix belongs and the field to which the target question belongs.

[0072] Optionally, the fingerprint features of each knowledge text in the designated database are determined by encrypting the knowledge text based on an information digest algorithm; before comparing the fingerprint features of the initial reply content with the fingerprint features of the extended knowledge text associated with each of the target keywords for similarity, the large language model also performs: encrypting the initial reply content based on the information digest algorithm to obtain the fingerprint features of the initial reply content.

[0073] Optionally, the first prompt word also includes vector features of professional knowledge text externally attached based on retrieval enhancement generation technology; wherein, if the professional knowledge text comes from the designated database, the fingerprint features of the initial reply content also include vector features of the external professional knowledge text.

[0074] Optionally, if the professional knowledge text comes from other databases, the large language model, before comparing the fingerprint features of the initial reply content with the fingerprint features of the extended knowledge text associated with each of the target keywords, further performs: obtaining the professional knowledge text from the other database based on the vector features of the professional knowledge text; encrypting the professional knowledge text based on the information summary algorithm to obtain the fingerprint features of the professional knowledge text; wherein the fingerprint features of the professional knowledge text belong to the fingerprint features of the initial reply content.

[0075] Optionally, the target question is asked by the communication user to the human-machine interaction robot of the communication operator, and the target reply content is the result of the human-machine interaction robot replying to the communication user.

[0076] It should be noted that the processing device for the large language model reply content of this embodiment can be used as Figure 1 The execution subject of the method shown can thus realize Figure 1 The steps and functions in the method shown.

[0077] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 7At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. Of course, the electronic device may also include hardware required for other services.

[0078] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0079] The memory is used to store the computer program. Specifically, the computer program may include a program code, and the program code includes a computer operation instruction. The memory may include a memory and a non-volatile memory, and provides the computer program to the processor. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming the above-mentioned logic layer. Figure 6 The processing device of the large language model reply content shown. Correspondingly, the processor executes the program stored in the memory and is specifically used to perform the following operations: After the large language model obtains the initial response content corresponding to the target question according to the first prompt word query, a second prompt word is constructed, and the second prompt word is used to guide the large language model to execute: determine the extended knowledge text related to the target question; and supplement the first prompt word based on the initial response content and the extended knowledge text.

[0080] The first prompt word and the second prompt word are input into the large language model, so that the large language model supplements the first prompt word according to the second prompt word, and obtains the target reply content corresponding to the target question based on the supplemented first prompt word.

[0081] After the user completes the initial query through the prompt word, the electronic device of this embodiment can use the large language model to intelligently supplement and optimize the prompt word according to the initial reply content found in the query, so as to query again based on the supplemented and optimized prompt word to improve the quality of the final reply content. Since the process of supplementing and optimizing the prompt word to re-query is completed on the deployment end of the large language model side, from the user's perception, it is equivalent to executing only one query, thereby increasing the probability that the first query can hit the user's expectations.

[0082] The above is as in this manual Figure 1 The processing method of the large language model reply content disclosed in the illustrated embodiment can be applied to a processor and implemented by the processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in a decoding processor. The software module can be located in a mature storage medium in the field such as random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0083] Of course, in addition to software implementation, the electronic device of this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0084] In addition, the present application also provides a computer program product, which includes a computer-readable storage medium, the computer-readable storage medium storing one or more computer programs, and the one or more computer programs including instructions. When the above instructions are executed by a portable electronic device including multiple application programs, the portable electronic device can execute Figure 1 The steps in the method shown include: After the large language model obtains the initial response content corresponding to the target question according to the first prompt word query, a second prompt word is constructed, and the second prompt word is used to guide the large language model to execute: determine the extended knowledge text related to the target question; and supplement the first prompt word based on the initial response content and the extended knowledge text.

[0085] The first prompt word and the second prompt word are input into the large language model, so that the large language model supplements the first prompt word according to the second prompt word, and obtains the target reply content corresponding to the target question based on the supplemented first prompt word.

[0086] It should be understood by those skilled in the art that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0087] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0088] The above are only embodiments of this specification and are not intended to limit this specification. For those skilled in the art, this specification may be subject to various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this specification shall be included in the scope of the claims of this specification. In addition, all other embodiments obtained by ordinary technicians in this field without creative work shall fall within the scope of protection of this document.

Claims

1. A method for processing reply content of a large language model, characterized in that: include: After the large language model obtains the initial response content corresponding to the target question according to the first prompt word query, a second prompt word is constructed, and the second prompt word is used to guide the large language model to perform: determining an extended knowledge text related to the target question; supplementing the first prompt word based on the initial response content and the extended knowledge text; The first prompt word and the second prompt word are input into the large language model, so that the large language model supplements the first prompt word according to the second prompt word, and obtains the target reply content corresponding to the target question based on the supplemented first prompt word.

2. The method according to claim 1, characterized in that The second prompt word is specifically used to guide the large language model to execute: determining whether the initial reply content meets the preset reply content requirements, and if the initial reply content does not meet the preset reply content requirements, determining the extended knowledge text related to the target question, and supplementing the first prompt word based on the initial reply content and the extended knowledge text.

3. The method according to claim 2, characterized in that The preset reply content requirement includes that the number of knowledge blocks of the first prompt word for the initial reply content reaches a preset block number threshold, and / or the number of words in the initial reply content reaches a preset word number threshold.

4. The method according to claim 1, characterized in that: The step of determining the extended knowledge text related to the target problem includes: determining at least one target keyword related to the target question; A knowledge text related to each target keyword is queried from a designated database as an extended knowledge text; wherein the designated database stores a plurality of knowledge texts and a correspondence between the plurality of knowledge texts and keywords.

5. The method according to claim 4, characterized in that The determining of at least one target keyword related to the target question includes: Performing intent recognition on the target question to obtain the question intent corresponding to the target question; Keyword extraction is performed on the question intention to obtain at least one target keyword related to the target question.

6. The method according to claim 4, characterized in that The designated database also stores the fingerprint feature of each knowledge text; the second prompt word is used as an appendix of the first prompt word, and the appendix includes a supplementary item set for each target associated word, and the supplementary item includes an extended knowledge text of the target associated word and the fingerprint feature of the extended knowledge text; The supplementing the first prompt word based on the initial reply content and the extended knowledge text includes: Comparing the fingerprint features of the initial reply content with the fingerprint features of the extended knowledge text associated with each target keyword for similarity to determine whether there is redundant extended knowledge text that is repeated with the initial reply content; If it is determined that there is a redundant extended knowledge text, the supplementary item corresponding to the redundant extended knowledge text in the appendix is ​​removed, so as to add the appendix after removal to the first prompt word; If it is determined that there is no redundant extended knowledge text, the appendix is ​​directly added to the first prompt word.

7. The method according to claim 6, characterized in that The second prompt word is also used to guide the large language model to execute: when there is no supplementary item in the appendix, directly use the initial response content as the query result of the target question.

8. The method according to claim 6, characterized in that The appendix also includes a fingerprint feature list, which includes fingerprint features of the extended knowledge text associated with each target associated word; The comparing the fingerprint feature of the initial reply content with the fingerprint feature of the extended knowledge text associated with each target keyword to determine whether there is redundant extended knowledge text that is repeated with the initial reply content includes: The fingerprint features of the initial reply content are compared with the fingerprint features recorded in the fingerprint feature list for similarity to determine whether there is redundant extended knowledge text that is repeated with the initial reply content.

9. The method according to claim 8, characterized in that The second prompt word is also used to guide the large language model to execute: deleting the fingerprint feature list in the appendix and the fingerprint features in the supplementary items, so as to use the remaining content in the appendix as part of the target reply content.

10. The method according to claim 9, characterized in that The second prompt word is also used to guide the large language model to execute: before deleting the fingerprint feature list in the appendix and the fingerprint features in the supplementary items, quality scoring the extended knowledge text of each supplementary item in the appendix, and eliminating the supplementary items in the appendix whose quality scores do not meet the preset scoring requirements.

11. The method according to claim 10, characterized in that The quality of the extended knowledge text of each supplement in the appendix was scored, including: Based on the similarity between the semantics of the extended knowledge text of each supplementary item in the appendix and the semantics of the target question, a quality score is given to the extended knowledge text of each supplementary item in the appendix; and / or, Based on the correlation between the field to which the extended knowledge text of each supplementary item in the appendix belongs and the field to which the target question belongs, a quality score is given to the extended knowledge text of each supplementary item in the appendix.

12. The method according to claim 6, characterized in that The fingerprint feature of each knowledge text in the specified database is determined by performing encryption calculation on the knowledge text based on the information digest algorithm; Before comparing the fingerprint features of the initial reply content with the fingerprint features of the extended knowledge text associated with each target keyword for similarity, the method further includes: The initial reply content is encrypted based on the information digest algorithm to obtain a fingerprint feature of the initial reply content.

13. The method according to claim 12, characterized in that The first prompt word also includes the vector features of the professional knowledge text based on the retrieval enhancement generation technology; wherein, if the professional knowledge text comes from the designated database, the fingerprint features of the initial reply content also include the vector features of the professional knowledge text.

14. The method according to claim 13, characterized in that If the professional knowledge text comes from other databases, before comparing the fingerprint features of the initial reply content with the fingerprint features of the extended knowledge text associated with each target keyword for similarity, the method further includes: Based on the vector features of the professional knowledge text, acquiring the professional knowledge text from the other database; The professional knowledge text is encrypted based on the information summary algorithm to obtain the fingerprint feature of the professional knowledge text; wherein the fingerprint feature of the professional knowledge text belongs to the fingerprint feature of the initial reply content.

15. The method according to any one of claims 1 to 14, characterized in that: The target question is raised by the communication user to the human-machine interaction robot of the communication operator, and the target reply content serves as the reply result of the human-machine interaction robot to the target question.

16. A device for processing large language model reply content, characterized in that: include: A prompt word adaptation module is used to construct a second prompt word after the large language model obtains the initial response content corresponding to the target question according to the first prompt word query, and the second prompt word is used to guide the large language model to execute: determining the extended knowledge text related to the target question; supplementing the first prompt word based on the initial response content and the extended knowledge text; The query supplement module is used to input the first prompt word and the second prompt word into the large language model so that the large language model supplements the first prompt word according to the second prompt word, and obtains the target reply content corresponding to the target question based on the supplemented first prompt word.

17. An electronic device comprising a processor; and a memory configured to store computer executable instructions, characterized in that: The computer executable instructions, when executed, cause the processor to perform the method of any one of claims 1-15.

18. A computer program product, comprising a computer-readable storage medium storing a computer program, characterized in that: The computer program is operable to cause a computer to perform the method according to any one of claims 1 to 15.

Images