A large model retrieval enhanced generation method and system based on diffusion alignment

Abstract

A diffusion alignment-based large model retrieval enhancement generation method and system are disclosed, which comprises obtaining user input text information, and using a vector model to encode the text information to obtain an encoded semantic vector; retrieving text information similar to the encoded semantic vector from a vector database, and selecting text information meeting the conditions according to a similarity threshold; performing entity extraction on the text information meeting the conditions, indexing the extracted entity information and the entity relationship stored in a graph database to generate enhanced background information containing additional semantic information; diffusing the enhanced background information from the original word vector space to the word vector space of a large language model to obtain a diffusion offset vector consistent with the semantic distribution of the language model; splicing the diffusion offset vector and the encoded vector of the user input text information, and inputting the same into the large language model for reasoning to output a generation result containing retrieval enhancement information. The present application can greatly improve the reasoning speed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of large model retrieval, and in particular to a large model retrieval enhancement generation method and system based on diffusion alignment. BACKGROUND

[0002] Retrieval-augmented Generation (RAG) takes advantage of the context learning ability of large language models to enable large language models to acquire the latest knowledge without training, avoiding the defect of acquiring the latest knowledge by retraining model parameters, greatly reducing the consumption of computing resources while ensuring that the model performance is not reduced. The usual implementation of RAG is to use a vector model to encode the latest knowledge, and then save the encoded information to a vector database; in the inference process, the user's question is encoded using a vector model, and then the encoded vector is calculated with the vector in the vector database. The similarity is usually calculated by Euclidean distance or cosine similarity. Then return the text information with high similarity as part of the model input, which alleviates the difficulty of not being able to answer new questions due to fixed model parameters.

[0003] One of the key points of RAG technology is to accurately and quickly recall relevant text information according to the user's question. Since the large language model needs to go through retrieval, similarity calculation and other steps before generation, the inference speed is greatly reduced compared to general methods; in addition, since the Embedding of the vector model is different from the Embedding of the large language model, the information considered relevant by the vector model may not be relevant in the large language model, that is, the retrieval model and the large language model are not aligned, there is a semantic understanding difference; even if the relevant information can be accurately recalled, it is impossible to eliminate the human error or data error brought about when building the database, which will seriously affect the output results of the large language model (hallucination, toxicity, instability, etc.). How to accurately recall relevant semantic information while ensuring inference speed is still a problem to be solved. SUMMARY

[0004] In order to solve the above technical problems in the prior art, the present application provides a large model retrieval enhancement generation method and system based on diffusion alignment to solve the above technical problems.

[0005] According to a first aspect of the present application, a large model retrieval enhancement generation method based on diffusion alignment is provided, comprising:

[0006] S1: Obtain the user's question text information, encode it using a vector model to obtain a semantic vector, calculate the cosine similarity between the semantic vector and the vectors obtained from the vector database in turn, and extract the text information and vector information with a similarity that meets the threshold;

[0007] S2: Perform entity extraction on the text information, define entity information categories and make them consistent with the definitions in the graph database, obtain an entity sequence, index the extracted entity information in the graph database, obtain additional semantic information and corresponding encoding vectors based on entities and entity relationships, use the encoding vectors as known information to obtain entity-enhanced graph database information, and combine the graph database information with the information in the vector database to obtain the final background information;

[0008] S3: Spread the final background vector information from the original BGE word vector space to the word vector space of the language model, introduce a preset parameter and calculate the spread offset vector;

[0009] S4: Continue to input the spread offset vector and the updated vector into the semantic spread structure, repeat S3 to obtain the information spread offset vector, subtract the spread offset vector from the information spread offset vector bit by bit to obtain the semantic vector that eliminates pseudo semantic noise, and cycle until the step number reaches the preset value to obtain the final spread offset vector;

[0010] S5: Encode the user's question text information using a model consistent with the language model to obtain a vector form, concatenate the spread offset vector with the vector form, obtain a three-dimensional semantic vector containing the user's question text information and recall information encoding information, and input the large language model to obtain the final output result.

[0011] In some specific embodiments, S1 specifically includes: obtaining the user's question text information Q = {q1, q2,..., qn}, encoding it using a BGE vector model to obtain a semantic vector E = BGE(Q) = {e1, e2,..., en}, and calculating the cosine similarity as n n where d n is the encoding form of the text information in the vector database, which is obtained by encoding the vector library text information.

[0012] In some specific embodiments, S2 specifically includes: obtaining the defined entity sequence where n n is the entity information sequence contained in a single information; indexing the extracted entity information from the graph database where the storage form of the graph database is a quadruple, i.e., m = (n n , r n , p n ​​, e n}, wherein n, r, p respectively represent entity information 1, the relationship between entities, and entity information 2; e represents a vector representation of text information between node n and node p; additional semantic information corresponding to the encoding vector of the sentence information is obtained according to the entity and the relationship between entities; the encoding vector is taken as known information to obtain entity-enhanced graph database information The graph database information G is combined with the information in the vector database to obtain final background information

[0013] In some specific embodiments, in S3, the final background vector information is diffused from the original BGE word vector space to the word vector space of the language model by using a diffusion model. The diffusion model performs multi-step diffusion on the final background information through a parameter matrix, and controls the diffusion stability through KL divergence and a multi-head attention mechanism: wherein, is the diffusion offset output at the last time, is used to approximate distribution.

[0014] In some specific embodiments, in the diffusion process in S3, hyperparameters a, b, and g are introduced to limit the amplitude of each diffusion step, wherein a+b+g=1; the vector output by the residual structure in the diffusion model is scaled, and the expression of the scaling operation is as follows wherein x is the vector output after the FFW module in the diffusion model, and x t is the vector output after the scaling operation; x t is input into the FFN network structure, and the vector output by the FFN network structure is offset by a to obtain the diffusion offset at the current time.

[0015] In some specific embodiments, in S5, the diffusion offset vector is spliced with the vector form, and the specific representation is as follows wherein B represents the batch input into the model, Seq represents the length received by the model, Dim represents the dimension of the word vector, represents the final diffusion offset vector, and E * represents the vector form of the encoded user information.

[0016] According to a second aspect of the present application, a computer readable storage medium is provided, which stores one or more computer programs, and the one or more computer programs are executed by a computer processor to implement the above method.

[0017] According to a third aspect of the present application, a large model retrieval enhancement generation system based on diffusion alignment is provided, which comprises:

[0018] The information extraction unit is configured to acquire the user's question text information, encode it using a vector model to obtain a semantic vector, calculate the cosine similarity between the semantic vector and the vectors sequentially obtained from the vector database, and extract text information and vector information whose similarity meets the threshold.

[0019] The background information acquisition unit is configured to extract entities from text information, define entity information categories and make them consistent with the definitions in the graph database, obtain entity sequences, index the extracted entity information in the graph database, obtain additional semantic information and corresponding encoding vectors based on entities and entity relationships, use the encoding vectors as known information to obtain entity-enhanced graph database information, and then merge the graph database information with the information in the vector database to obtain the final background information.

[0020] The diffusion offset unit is configured to diffuse the final background vector information from the original BGE word vector space to the word vector space of the language model. It introduces preset parameters and calculates the diffusion offset vector. The diffusion offset vector and the updated vector are then input into the semantic diffusion structure. The diffusion operation is repeated to obtain the information diffusion offset vector. The diffusion offset vector is subtracted from the information diffusion offset vector bit by bit to obtain the semantic vector that eliminates pseudo-semantic noise. The process is repeated until the number of steps reaches the preset value to obtain the final diffusion offset vector.

[0021] The result output unit is configured to encode the user's question text information into a vector form using a model consistent with the language model, concatenate the diffusion offset vector with the vector form, and obtain a three-dimensional semantic vector containing the user's question text information and recall information encoding information. This vector is then input into the large language model for inference to obtain the final output result.

[0022] In some specific embodiments, the information extraction unit specifically includes obtaining the user's question text information Q = {q1, q2, ..., q...} n The semantic vector E = BGE(Q) = {e1, e2, ..., e} is encoded using the BGE vector model. n The cosine similarity is calculated as follows: Where, d n It is the encoding format of text information in the vector database, which is obtained by encoding the text information in the vector database.

[0023] In some specific embodiments, the background information acquisition unit obtains the defined entity sequence. Where n n This refers to the sequence of entity information contained in a single message; the extracted entity information is indexed from the graph database. The graph database is stored in the form of quadruples, i.e., m = (n n r n pn , e n ), wherein n, r, p respectively represent entity information 1, the relationship between entities, and entity information 2; e represents a vector representation of text information between nodes n and p; additional semantic information corresponding to the coding vector of the sentence information is obtained according to the entity and the relationship between entities; the coding vector is taken as known information to obtain entity-enhanced graph database information The graph database information G is combined with the information in the vector database to obtain final background information

[0024] In some specific embodiments, the diffusion offset unit utilizes a diffusion model to diffuse the final background vector information from the original BGE word vector space to the word vector space of the language model, the diffusion model performs multi-step diffusion on the final background information through a parameter matrix, and the diffusion stability is controlled through KL divergence and a multi-head attention mechanism: wherein, is the diffusion offset output at the previous time, is used to approximate distribution; in the diffusion process in the diffusion offset unit, hyperparameters alpha, beta and gamma are introduced to limit the amplitude of each step of diffusion, wherein alpha+beta+gamma=1; the vector output by the residual structure in the diffusion model is subjected to a scaling operation, and the expression of the scaling operation is as follows wherein chi is the vector output after the FFW module in the diffusion model, X t is the vector output after the scaling operation; X t is input into the FFN network structure, and the vector output by the FFN network structure is subjected to alpha offset to obtain the diffusion offset at the current time.

[0025] In some specific embodiments, the diffusion offset vector is spliced with the vector form in the result output unit, and the specific representation is as follows wherein B represents Batch input into the model, Seq represents the length received by the model, Dim represents the dimension of the word vector, represents the final diffusion offset vector, E * represents the vector form of the encoded user information.

[0026] The present application proposes a large model retrieval enhancement generation method and system based on diffusion alignment, which maps the semantic distribution of the vector retrieval model into the language model, and then embeds the semantic features of the vector into the encoded text vector, avoiding re-encoding of the text and greatly improving the reasoning speed of the model. At the same time, the present application combines entity recognition technology and graph database indexing, greatly improves the breadth and accuracy of recall information, and reduces the illusion, toxicity and instability of the model. BRIEF DESCRIPTION OF DRAWINGS

[0027] The accompanying drawings are included to provide a further understanding of embodiments and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments and, together with the description, serve to explain the principles of the application. Other embodiments and many of the intended advantages of the present application will be readily appreciated as the same becomes better understood by reference to the following detailed description when considered in connection with the accompanying drawings:

[0028] Figure 1 is a flow chart of a diffusion alignment based large model retrieval enhanced generation method of an embodiment of the present application;

[0029] Figure 2 is a flow chart of a diffusion alignment based large model retrieval enhanced generation method of a specific embodiment of the present application;

[0030] Figure 3 is a retrieval retrieval structure diagram of a specific embodiment of the present application;

[0031] Figure 4 is a distribution diffusion framework diagram of a specific embodiment of the present application;

[0032] Figure 5 is a diffusion model structure diagram of a specific embodiment of the present application;

[0033] Figure 6 is a diffusion semantic embedding schematic diagram of a specific embodiment of the present application;

[0034] Figure 7 is a semantic distribution change diagram in the diffusion model training process of a specific embodiment of the present application;

[0035] Figure 8 is a diffusion alignment based large model retrieval enhanced generation system architecture diagram of an embodiment of the present application;

[0036] Figure 9 is a structural schematic diagram of a computer system of an electronic device suitable for implementing embodiments of the present application. DETAILED DESCRIPTION

[0037] The present application will be further described below in conjunction with the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the related application, and not to limit the application. In addition, it should be noted that only the parts related to the application are shown in the drawings for ease of description.

[0038] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other in the case of no conflict. The present application will be described in detail below with reference to the drawings and in combination with the embodiments.

[0039] Figure 1 A flow chart of a large model retrieval enhanced generation method based on diffusion alignment according to an embodiment of the present application is shown. As shown in Figure 1 the method comprises the following steps:

[0040] S1: obtaining the user's question text information, encoding the question text information using a vector model to obtain a semantic vector, calculating the cosine similarity of the semantic vector and the vectors obtained from the vector database in turn, and extracting the text information and vector information with a similarity meeting a threshold.

[0041] In a specific embodiment, the user's question text information Q {q1, q2,.., q n is obtained, the BGE vector model is used to encode the question text information to obtain a semantic vector E = BGE(Q) = {e1, e2,.., e n , and the cosine similarity calculation method is where d n is the encoding form of the text information in the vector database, which is obtained by encoding the vector library text information.

[0042] S2: performing entity extraction on the text information, defining entity information categories and making them consistent with the definitions in the graph database, obtaining an entity sequence, indexing the extracted entity information in the graph database, obtaining additional semantic information and corresponding encoding vectors according to the entities and entity relationships, obtaining entity-enhanced graph database information using the encoding vectors as known information, and merging the graph database information with the information in the vector database to obtain the final background information.

[0043] In a specific embodiment, the defined entity sequence is obtained, where n n is the sequence of entity information contained in a single piece of information; the extracted entity information is indexed from the graph database where the storage form of the graph database is a quadruple, i.e., m = (n n , r n , p n , e n ), where n, r, and p represent entity information 1, the relationship between entities, and entity information 2, respectively; e represents the vector representation of the text information between node n and node p; additional semantic information and the encoding vector corresponding to the sentence information are obtained according to the entities and the relationships between the entities; entity-enhanced graph database information is obtained using the encoding vector as known information Merge the graph database information G with the information in the vector database to obtain final background information

[0044] S3: Spread the final background vector information from the original BGE word vector space to the word vector space of the language model, introduce a preset parameter, and calculate a spread offset vector.

[0045] In specific embodiments, the final background vector information is spread from the original BGE word vector space to the word vector space of the language model using a spread model. The spread model performs multi-step spreading on the final background information through a parameter matrix, and controls the stability of the spreading through KL divergence and a multi-head attention mechanism: wherein, is the spread offset output at the previous time, is used to approximate the distribution.

[0046] S4: Continue to input the spread offset vector and the updated vector into the semantic spread structure, repeat S3 to obtain an information spread offset vector, subtract the spread offset vector from the information spread offset vector bit by bit to obtain a semantic vector that eliminates pseudo semantic noise, and cycle until the step number reaches a preset value to obtain a final spread offset vector.

[0047] In specific embodiments, during the above spread process, hyperparameters α, β, and γ are introduced to limit the amplitude of each step of spreading, where α+β+γ=1; the vector output by the residual structure in the spread model is scaled, and the expression of the scaling operation is as follows wherein χ is the vector output after the FFW module in the spread model, X t is the vector output after the scaling operation; X t is input into the FFN network structure, and the vector output by the FFN network structure is offset by α to obtain the spread offset at the current time.

[0048] S5: Encode the user's question text information into a vector form using a model consistent with the language model, concatenate the spread offset vector with the vector form to obtain a three-dimensional semantic vector containing the user's question text information and recall information encoding information, and input the three-dimensional semantic vector into a large language model to obtain a final output result.

[0049] In specific embodiments, concatenating the spread offset vector with the vector form is specifically represented as wherein B represents the Batch input into the model, Seq represents the length received by the model, Sim represents the dimension of the word vector, represents the final spread offset vector, E * represents the vector form of the encoded user information.

[0050] In specific embodiments, Figure 2 A flowchart of the diffusion alignment-based large model retrieval enhancement generation method of one specific embodiment of the application is shown in FIG. 1, which includes the following steps: Figure 2 As shown in FIG. 1, the steps include:

[0051] Step 1: The user inputs a natural language question; the retrieval recall technology is used to obtain the vector representation of the relevant text information.

[0052] In specific embodiments, Figure 3 A retrieval recall structure diagram of one specific embodiment of the application is shown in FIG. 2, which includes the following steps: Figure 3 As shown in FIG. 2, the specific process of retrieval and recall in the vector database includes:

[0053] After the user input text is encoded into a semantic vector, similarity calculation is performed with the stored vectors in the database.

[0054] The vectors that meet the conditions are screened out by setting a similarity threshold.

[0055] The screened vectors represent the most relevant background information to the user's input question, which are used as the input for the next step.

[0056] In specific embodiments, the user's question (text information) Q = {q1, q2,.., q n} is obtained. The text information is encoded using a BGE vector model to obtain the encoded semantic vector E = BGE(Q) = {e1, e2,.., e n}. After obtaining the encoded semantic vector, the vectors are obtained from the vector database in sequence, and similarity calculation is performed with the encoded semantic vector. The calculation method is cosine similarity:

[0057] where d n is the encoded form of the text information in the vector database, which is obtained by encoding the vector library text information T = {t1, t2,.., t n}, D = BGE(T) = {d1, d2,.., d n}. After calculating the similarity between the user's text semantic and the background knowledge in the database, the text information and vector information that meet the threshold are extracted, where σ is a preset threshold. When D recall meets the threshold requirement, the entity extraction of the second step is performed, otherwise the next information is obtained.

[0058] Step 2: The text information is subjected to entity extraction to obtain an entity sequence and is indexed in the graph database to obtain a quadruple. The information in the graph database is obtained and merged with the retrieval recall information.

[0059] In a specific embodiment, the text containing relevant background information obtained in the first step is used for entity extraction. The entity information to be identified is defined, and the defined entity categories must be consistent with those defined in the graph database. This results in a sequence of defined entities. Where n n This refers to the sequence of entity information contained in a single message. The extracted entity information is indexed from the graph database. In this context, the graph database is stored in the form of quadruples, i.e., m = (n... n r n p n e n ), where n, r, and p represent entity information 1, the relationship between entities, and entity information 2, respectively; e represents the vector representation of the text information between node n and node p. Based on the entities and the relationships between them, additional semantic information and the encoding vector corresponding to the sentence information are obtained; using the encoding vector as known information, the entity-enhanced graph database information G is obtained: The information from the graph database G is merged with the information from the vector database to obtain the final background information.

[0060] In a specific embodiment, Figure 7 This application illustrates a semantic distribution change graph during the training process of a diffusion model according to a specific embodiment of the present application, as shown below. Figure 7 As shown in the figure, a to c represent the process of the model semantic distribution gradually spreading to the vector semantic distribution during training. The difference between the vector semantic distribution encoded by the retrieval model and the semantic distribution of the language model can be regarded as semantic noise. During training, the vector semantic distribution gradually spreads to the model semantic distribution, thereby avoiding the problem of secondary encoding of text information by the model.

[0061] Step 3: The semantic diffusion model diffuses relevant information from the vector semantic space to the large model semantic space.

[0062] In a specific implementation, the merged vector information is diffused from the original BGE word vector space to the word vector space of the language model: Where t is a manually set hyperparameter, representing the number of steps in the diffusion process. This refers to the background information in the vector representation obtained in step three. The specific diffusion model is as follows: Figure 5 As shown: where the coefficient matrix is ​​the parameter matrix, t=1 is set as the initial value, and one-hot encoding is performed on t to obtain the encoded vector representation. k is the total number of diffusion steps. Combine the parameter matrix and... After the product is carried out, the pseudo semantic noise vector corresponding to the time t is obtained, in order to facilitate the subsequent embedding into the vector representation of the semantic model, the k-dimensional vector needs to be mapped to the same dimension dim as the model: Figure 5 Token Embedding in is the vector representation form of the background information in step two In order to ensure the stability in the diffusion process, the application introduces three preset hyperparameters (alpha, beta, gamma) to limit the amplitude of each step of diffusion, wherein alpha+beta+gamma=1. The design of the FFW module is to further constrain the diffusion offset, and the specific operation is as follows: in order to ensure the true effectiveness of the diffusion process, it is necessary to ensure that the newly obtained diffusion offset vector is similar to the diffusion offset vector at the last time point in the diffusion process. The difference between the new diffusion offset vector obtained by each diffusion and the diffusion offset vector obtained by the last step is measured by KL divergence: Wherein, is the diffusion offset vector output at the last time point, and is used to approximate the distribution of , when the approximation degree is less than the set threshold value, the regularization processing needs to be performed on , is the vector output after the multi-head attention mechanism.

[0063] In specific embodiments, Scale represents the scaling operation on the vector output by the residual structure, and the specific expression is as follows: Wherein, chi is the vector output after FFW, and X t is the vector output after the Scale operation, and after the Scale operation, the variance of the distribution of X t does not change, and the mean value becomes X t After obtaining X Factor Shift represents the alpha offset of the vector information output by the FFN, and the specific expression is as follows: After the Factor Shift, the diffusion offset vector at the current time is obtained, and its distribution can be approximated to a normal distribution: Wherein, mu t ∝alpha, and V t-1 ∝beta.

[0064] In specific embodiments, the diffusion offset vector V obtained above={v1, v2,..., v n} and the updated T=t+1 are continuously input into the semantic diffusion structure (as shown in Figure 4 ), and the information diffusion offset vector V is obtained by repeating the foregoing operation The semantic vector eliminating pseudo semantic noise is obtained by subtracting the diffusion offset vector V obtained in the previous step from the offset vector obtained in this diffusion step by bit by bit This operation is repeated until the number of steps reaches the set value, i.e., T=k. When the cycle reaches the specified number of steps, the final diffusion offset vector is obtained

[0065] Step four, the large language model encodes the user's question, and the output result is a word vector; the vector representation of the semantic space is spliced with the user question encoding; and the large model is embedded and forward propagated.

[0066] In a specific embodiment, the user information Q of the first step is text encoded, and the Embedding model used is consistent with the language model (in this application, Llama is taken as an example, but this method can be applied to any Decoder-Only large language model). The vector form of the encoded user information is obtained The diffusion offset vector obtained in the third step is spliced with E * , as shown in Figure 6 , to obtain the spliced vector representation. This vector representation contains both the user's text information and the encoded information of the recall information, without the need to encode the recalled text information, and the vector representation is consistent with the distribution of the large language model in the semantic space after the semantic mapping of the diffusion model in step three. Figure 6 The Text-Embedding in * is E in this step, and the K-Embedding is the final diffusion offset vector obtained in the fourth step The splicing expression is Where B represents the Batch of the input model, Seq represents the length received by the model, and Dim represents the dimension of the word vector. In particular, the splicing is performed from the Seq dimension, and the spliced vector is a three-dimensional semantic vector. After obtaining the spliced vector , the final output result is obtained through the reasoning of the large language model.

[0067] The application is directed to a large language model retrieval recall generation RAG, and a large language model retrieval embedding generation method based on diffusion alignment is designed. Since the inconsistency of semantic distribution between vector models and language models is caused by the error of the model itself, this error is closely related to the pre-training data and model structure of the model. Based on this assumption, a semantic diffusion model is trained to fill in this error, map the semantic distribution of the vector retrieval model into the language model, and then embed the semantic features of the vector into the encoded text vector, avoiding re-encoding of the text and greatly improving the reasoning speed of the model. At the same time, the invention combines entity recognition technology and graph database indexing to greatly improve the breadth and accuracy of the recall information and reduce the illusion, toxicity and instability of the model.

[0068] Figure 8 The application shows a large model retrieval enhancement generation system architecture diagram based on diffusion alignment of one embodiment of the application, as shown in Figure 8 The system includes an information extraction unit 801, a background information acquisition unit 802, a diffusion offset unit 803 and a result output unit 804, wherein the information extraction unit 801 is configured to acquire the user's question text information, encode it using a vector model to obtain a semantic vector, calculate the cosine similarity of the semantic vector with the vectors obtained from the vector database in turn, and extract the text information and vector information with a similarity that meets the threshold; the background information acquisition unit 802 is configured to perform entity extraction on the text information, define entity information categories and make them consistent with the definitions in the graph database, obtain an entity sequence, index the extracted entity information in the graph database, obtain additional semantic information and corresponding encoding vectors according to the entities and entity relationships, obtain entity-enhanced graph database information using the encoding vectors as known information, and then combine the graph database information with the information in the vector database to obtain the final background information; the diffusion offset unit 803 is configured to diffuse the final background vector information from the original BGE word vector space to the word vector space of the language model, introduce a preset parameter and calculate to obtain a diffusion offset vector; continue to input the diffusion offset vector and the updated vector into the semantic diffusion structure, repeat the diffusion operation to obtain an information diffusion offset vector, subtract the diffusion offset vector from the information diffusion offset vector bit by bit to obtain a semantic vector that eliminates pseudo semantic noise, and cycle until the step number reaches the preset value to obtain the final diffusion offset vector; the result output unit 804 is configured to encode the user's question text information using a model consistent with the language model to obtain a vector form, splice the diffusion offset vector with the vector form to obtain a three-dimensional semantic vector containing the user's question text information and recall information encoding information, and input the three-dimensional semantic vector into a large language model to obtain a final output result.

[0069] The following refers to Figure 9It shows a schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application. Figure 9 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0070] like Figure 9 As shown, the computer system includes a central processing unit (CPU) 901, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 902 or programs loaded from storage section 908 into random access memory (RAM) 903. RAM 903 also stores various programs and data required for the operation of system 900. CPU 901, ROM 902, and RAM 903 are interconnected via bus 904. Input / output (I / O) interface 905 is also connected to bus 904.

[0071] The following components are connected to I / O interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a liquid crystal display (LCD) and speakers, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card and a modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to I / O interface 905 as needed. A removable medium 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 910 as needed so that computer programs read from it can be installed into storage section 908 as needed.

[0072] In particular, the processes described above with reference to the flow charts can be implemented as a computer software program according to embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable storage medium, the computer program comprising program code for performing the methods illustrated by the flow charts. In such embodiments, the computer program can be downloaded and installed from a network via the communication section 909, and / or installed from the removable media 911. When the computer program is executed by the central processing unit (CPU) 901, the above-described functions defined in the methods of the present application are performed. It should be noted that the computer readable storage medium of the present application can be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium may, for example, be - but is not limited to - an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus or device. In the present application, the computer readable signal medium can include a data signal carried in a baseband or as part of a carrier wave, in which the computer readable program code is carried. Such a propagated data signal can take any of a variety of forms, including but not limited to electro-magnetic, optical, or any suitable combination thereof. The computer readable signal medium can also be any computer readable storage medium that can be used for by or in connection with an instruction execution system, apparatus or device, and that can contain or store a program for use by or in connection with an instruction execution system, apparatus or device. The program code contained on the computer readable storage medium can be transmitted by any suitable medium, including but not limited to wireless, wire line, optical fiber cable, RF, etc., or any suitable combination of the above.

[0073] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0074] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0075] The modules involved in the embodiments of the present application can be implemented in the form of software or hardware.

[0076] As another aspect, the present application also provides a computer readable storage medium, which can be included in the electronic device described in the above embodiments, or can exist independently without being assembled into the electronic device. The computer readable storage medium carries one or more programs, when the one or more programs are executed by the electronic device, the electronic device is caused to: acquire text information input by a user, and encode the text information by using a vector model to obtain an encoded semantic vector; retrieve text information similar to the encoded semantic vector from a vector database, and select text information meeting a condition according to a similarity threshold; perform entity extraction on the text information meeting the condition, index extracted entity information and entity relationships stored in a graph database, and generate enhanced background information containing additional semantic information; diffuse the enhanced background information from an original word vector space to a word vector space of a large language model to obtain a diffusion offset vector consistent with a semantic distribution of the language model; splice the diffusion offset vector and the encoded vector of the text information input by the user, and input the diffusion offset vector and the encoded vector into the large language model for reasoning, and output a generation result containing retrieval enhanced information.

[0077] The above description is merely the preferred embodiments of the present application and the explanation of the applied technical principles. It should be understood by those skilled in the art that the inventive scope involved in the present application is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by any combinations of the above technical features or equivalent features without departing from the inventive concept. For example, the above features can be replaced with the technical features disclosed in the present application (but not limited to) having similar functions to form technical solutions.

Claims

1. A diffusion alignment-based large model retrieval enhanced generation method, characterized in that, Comprising: S1: obtaining the user's question text information, using a vector model to encode it to get a semantic vector, calculating the cosine similarity of the semantic vector and the vectors obtained from the vector database in turn, extracting the text information and vector information with similarity meeting the threshold; S2: performing entity extraction on the text information, defining entity information categories and making them consistent with the definitions in the graph database, obtaining entity sequences, indexing the extracted entity information in the graph database, obtaining additional semantic information and corresponding encoding vectors according to entities and entity relationships, using the encoding vectors as known information to obtain entity-enhanced graph database information, and merging the graph database information with the information in the vector database to obtain the final background vector information; S3: spreading the final background vector information from the original BGE word vector space to the word vector space of the language model, introducing a preset parameter and calculating to obtain a diffusion offset vector; S4: inputting the diffusion offset vector and the updated vector into the semantic diffusion structure again, repeating S3 to obtain an information diffusion offset vector, subtracting the diffusion offset vector from the information diffusion offset vector bit by bit to obtain a semantic vector that eliminates pseudo semantic noise, and repeating the cycle until the step number reaches a preset value to obtain the final diffusion offset vector; S5: using a model consistent with the language model to encode the user's question text information into a vector form, concatenating the diffusion offset vector with the vector form to obtain a three-dimensional semantic vector containing the user's question text information and recall information encoding information, and inputting the three-dimensional semantic vector into a large language model to obtain a final output result.

2. The diffusion alignment based large model retrieval enhanced generation method according to claim 1, characterized in that, The S1 specifically comprises: acquiring question text information of the user , and encoding the question text information by using a BGE vector model to obtain a semantic vector , wherein a cosine similarity calculation mode is , wherein, is an encoded form of text information in a vector database, the encoded form is obtained by encoding the vector library text information, and dim represents a dimension of the vector.

3. The diffusion alignment based large model retrieval enhanced generation method according to claim 1, characterized in that, The S2 specifically includes: obtaining the defined entity sequence , wherein is an entity information sequence contained in a single piece of information; indexing the extracted entity information from the graph database , wherein the storage form of the graph database is a quadruple, that is , wherein respectively represent entity information 1, the relationship between entities, and entity information 2. represents a node and the vector representation of the text information between the node ; according to the entity and the relationship between the entities, additional semantic information is obtained and a coding vector corresponding to the semantic information is obtained; the coding vector is taken as known information to obtain entity-enhanced graph database information , the graph database information is merged with the information in the vector database to obtain the final background information .

4. The diffusion alignment based large model retrieval enhanced generation method according to claim 3, characterized in that, The S3 diffuses the final background vector information from the original BGE word vector space to the word vector space of the language model by using a diffusion model, the diffusion model performs multi-step diffusion on the final background information through a parameter matrix, and the diffusion stability is controlled through KL divergence and a multi-head attention mechanism: wherein, is a diffusion offset output at the previous time, is used to approximate a distribution, represents the dimension of the word vector.

5. The diffusion alignment based large model retrieval enhanced generation method according to claim 4, characterized in that, In the diffusion process in S3, super parameters a, b and g are introduced to limit the amplitude of each step of diffusion, wherein, ; and performing a scaling operation on the vector output by the residual structure in the diffusion model, and the expression of the scaling operation is as follows , wherein, is the vector output after the FFW module in the diffusion model, is the vector output after the scaling operation; and is input into the FFN network structure, and a vector output by the FFN network structure is subjected to a a offset to obtain a diffusion offset at the present moment.

6. The diffusion alignment based large model retrieval enhanced generation method according to claim 5, characterized in that, The concatenation of the diffusion bias vector with the vector form in S5 is specifically represented as wherein, Batch representing the input model, Length representing the length received by the model, Dimension representing the dimension of the word vector, Final diffusion bias vector representing the final diffusion bias vector, Encoded user information vector form representing the encoded user information vector form.

7. A computer readable storage medium having stored thereon one or more computer programs. The one or more computer programs, when executed by a computer processor, implement the method of any one of claims 1-6.

8. A diffusion alignment based large model retrieval enhanced generation system, characterized in that, Comprising: An information extraction unit configured to obtain the user's question text information, encode it using a vector model to obtain a semantic vector, calculate the cosine similarity of the semantic vector and the vectors obtained from the vector database in turn, and extract the text information and vector information with similarity meeting the threshold; A background information acquisition unit configured to perform entity extraction on the text information, define entity information categories and make them consistent with the definitions in the graph database, obtain entity sequences, index the extracted entity information in the graph database, obtain additional semantic information and corresponding encoding vectors according to entities and entity relationships, use the encoding vectors as known information to obtain entity-enhanced graph database information, and merge the graph database information with the information in the vector database to obtain the final background vector information; A diffusion offset unit configured to spread the final background vector information from the original BGE word vector space to the word vector space of the language model, introduce a preset parameter and calculate to obtain a diffusion offset vector; continue to input the diffusion offset vector and the updated vector into the semantic diffusion structure, repeat the diffusion operation to obtain an information diffusion offset vector, subtract the diffusion offset vector from the information diffusion offset vector bit by bit to obtain a semantic vector that eliminates pseudo semantic noise, and repeat the cycle until the step number reaches a preset value to obtain the final diffusion offset vector; The result output unit is configured to encode the user's question text information into a vector form by using a model consistent with a language model, splice the diffusion offset vector and the vector form, obtain a three-dimensional semantic vector containing the user's question text information and recall information encoding information, and input a large language model to obtain a final output result.

9. The diffusion alignment based large model retrieval enhanced generation system of claim 8, wherein, The information extraction unit specifically comprises obtaining the user's question text information , and a BGE vector model is used to encode the question text information to obtain a semantic vector , the cosine similarity calculation mode is , wherein, is an encoded form of the text information in the vector database, the encoded form is obtained by encoding the vector library text information, and dim represents the dimension of the vector.

10. The diffusion alignment based large model retrieval enhanced generation system of claim 8, wherein, The entity sequence defined in the background information obtaining unit Wherein The entity information sequence contained in a single piece of information; index the extracted entity information from the graph database Wherein the storage form of the graph database is a quadruple, that is Wherein Respectively represent entity information 1, the relationship between entities, and entity information 2. Indicate nodes And the vector representation of the text information between nodes According to the entity and the relationship between entities, additional semantic information is obtained, and a code vector corresponding to the semantic information is obtained; the code vector is taken as known information to obtain entity-enhanced graph database information The graph database information Is merged with the information in the vector database to obtain the final background information .

11. The diffusion alignment based large model retrieval enhanced generation system of claim 10, wherein, The diffusion offset unit utilizes a diffusion model to diffuse the final background vector information from the original BGE word vector space to the word vector space of the language model, the diffusion model performs multi-step diffusion on the final background information through a parameter matrix, and the diffusion stability is controlled through KL divergence and a multi-head attention mechanism: wherein, is a diffusion offset output at a previous time, is a distribution for approximating , represents the dimension of the word vector; in the diffusion process in the diffusion offset unit, hyperparameters α, β, γ are introduced to limit the amplitude of each step of diffusion, wherein, ; a scaling operation is performed on the vector output by the residual structure in the diffusion model, and the expression of the scaling operation is as follows wherein, is the vector output after the FFW module in the diffusion model, is the vector output after the scaling operation; and is input into the FFN network structure, and the vector output by the FFN network structure is subjected to α offset to obtain the diffusion offset at the current time.

12. The diffusion alignment based large model retrieval enhanced generation system of claim 11, wherein, The concatenation of the diffusion offset vector and the vector form in the result output unit is specifically represented as wherein, Batch representing input model, Length representing length received by model, Dimension representing dimension of word vector, Final diffusion offset vector, Vector form representing encoded user information.

Images