Encryption enhancement generation method and system applying large language model, and program product

Abstract

The invention provides an encryption enhancement generation method and system applying a large language model and a program product, and relates to the technical field of large language models.The encryption enhancement generation method comprises the steps that based on demand information input by a user, a first encryption vector corresponding to the demand information is obtained through a preset encryption model; performing similarity degree matching on the first encryption vector and the encryption knowledge vector to determine the encryption knowledge vector matched with the first encryption vector as a second encryption vector; on the basis of the second encryption vector and a user key corresponding to the user, obtaining professional knowledge information corresponding to the second encryption vector through a decryption model corresponding to the user key; and based on the professional knowledge information and the demand information, generating a feedback result corresponding to the demand information through a large language model.

Description

Technical Field

[0001] The present invention relates to the technical field of large language models, and particularly to an encryption-enhanced generation method, system, and program product using a large language model. Background Art

[0002] A large language model (LLM) can be a type of natural language processing model based on deep learning that has been trained on a large amount of text / image data and is capable of processing and generating natural language text / images. With the continuous development and progress of large language model technology, it has played an indispensable and important role in more and more professional research fields.

[0003] Considering that there is a large amount of professional knowledge and academic terms in professional research fields, in order to avoid the problem of processing hallucination when the large language model processes professional knowledge in professional research fields, that is, the content generated by the large language model may seem reasonable on the surface but is actually fictional or inaccurate information, an enhanced generation knowledge base containing professional knowledge information can be connected to the large language model to overcome this problem. However, the professional knowledge information stored in the enhanced generation knowledge base is prone to the problem of professional knowledge information leakage during the process of assisting the large language model in content generation; in addition, the professional knowledge information contained in the enhanced generation knowledge base may also contain classified content that needs to be kept confidential, and it is also prone to being illegally obtained or leaked during the process of assisting the large language model in content generation.

[0004] In view of this, some embodiments of this specification provide an encryption-enhanced generation method, system, and program product using a large language model, which can ensure the confidentiality and security of professional knowledge information during the process of assisting the large language model in content generation. Summary of the Invention

[0005] Some embodiments of this specification provide an encryption-enhanced generation method using a large language model, and the method includes: based on the requirement information input by the user, obtaining a first encryption vector corresponding to the requirement information through a preset encryption model; performing a similarity degree matching between the first encryption vector and an encryption knowledge vector to determine the encryption knowledge vector that matches the first encryption vector as a second encryption vector; based on the second encryption vector and the user key corresponding to the user, obtaining the professional knowledge information corresponding to the second encryption vector through a decryption model corresponding to the user key; and generating a feedback result corresponding to the requirement information through the large language model based on the professional knowledge information and the requirement information.

[0006] In some embodiments, the encrypted knowledge vector is obtained by performing vectorized encryption processing on professional knowledge information stored in an enhanced generation knowledge base associated with a large language model through an encryption model; wherein, the encrypted knowledge vector is stored independently outside the enhanced generation knowledge base.

[0007] In some embodiments, the method further includes: verifying the legitimacy of a large language model generation request input by a user based on a preset license key, where the large language model generation request includes a user key and requirement information; and obtaining a first encrypted vector corresponding to the requirement information through an encryption model when the legitimacy verification passes.

[0008] In some embodiments, matching the similarity degree between the first encrypted vector and the encrypted knowledge vector to determine the encrypted knowledge vector that matches the first encrypted vector as the second encrypted vector includes: obtaining the cosine similarity between the first encrypted vector and the encrypted knowledge vector; and determining the encrypted knowledge vector whose cosine similarity belongs to a preset range as the second encrypted vector.

[0009] In some embodiments, the preset process of the encryption model includes: performing full-parameter adjustment training on a preset natural language processing model to make the input data distribution and output data distribution of the natural language processing model consistent; and determining the part before the vector representation stage in the natural language processing model after full-parameter adjustment training as the encryption model.

[0010] In some embodiments, the preset process of the decryption model includes: introducing a randomly initialized user feature vector into the part after the vector representation stage in the natural language processing model after full-parameter adjustment and performing local parameter adjustment training to make the input data distribution and output data distribution of the natural language processing model consistent; determining the part after the vector representation stage in the natural language processing model after local parameter adjustment training as the decryption model, where the decryption model corresponds one-to-one with the user feature vector; and using the user feature vector as the user key.

[0011] One or more embodiments of this specification also provide an encrypted enhanced generation system applying a large language model, where the system includes: a first server, configured to obtain a first encrypted vector corresponding to requirement information through a preset encryption model based on the requirement information input by a user; and configured to match the similarity degree between the first encrypted vector and the encrypted knowledge vector to determine the encrypted knowledge vector that matches the first encrypted vector as the second encrypted vector and output it; a second server, configured to obtain professional knowledge information corresponding to the second encrypted vector through a decryption model corresponding to the user key based on the second encrypted vector and the user key corresponding to the user; and the large language model is carried on the second server and is configured to generate a corresponding feedback result based on the professional knowledge information and the requirement information.

[0012] In some embodiments, the system further includes: an enhanced generation knowledge base for storing professional knowledge information; the first server includes an encrypted knowledge vector library, which is used to obtain and store encrypted knowledge vectors through an encryption model based on the professional knowledge information stored in the enhanced generation knowledge base; the first server and the enhanced generation knowledge are separated from each other and are independently deployed.

[0013] One or more embodiments of this specification also provide an encryption-enhanced generation system using a large language model. The system includes: a requirement encryption module for obtaining a first encrypted vector corresponding to the requirement information through a preset encryption model based on the requirement information input by the user; a knowledge matching module for matching the similarity between the first encrypted vector and the encrypted knowledge vectors to determine the encrypted knowledge vector that matches the first encrypted vector as the second encrypted vector; a knowledge decryption module for obtaining the professional knowledge information corresponding to the second encrypted vector through a decryption model corresponding to the user key based on the second encrypted vector and the user key corresponding to the user; a requirement feedback module for generating a feedback result corresponding to the requirement information through the large language model based on the professional knowledge information and the requirement information.

[0014] Some embodiments of this specification also provide a computer program product, including computer instructions or computer programs. When at least part of the computer instructions or computer programs are executed by a processor, they can implement the encryption-enhanced generation method using a large language model provided in any embodiment of this specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] This specification will be further described in the form of exemplary embodiments, and these exemplary embodiments will be described in detail through the drawings. The same numbers in the drawings represent the same structures or steps.

[0016] Figure 1 is a schematic structural diagram of a requirement feedback system using a large language model shown in some embodiments of this specification.

[0017] Figure 2 is an exemplary flowchart of an encryption-enhanced generation method using a large language model shown in some embodiments of this specification.

[0018] Figure 3 is an exemplary flowchart of a preset process of an encryption model shown in some embodiments of this specification.

[0019] Figure 4 is an exemplary flowchart of a preset process of a decryption model shown in some embodiments of this specification.

[0020] Figure 5It is a schematic data flow diagram of an encryption-enhanced generation system applying a large language model as shown in some embodiments of this specification.

[0021] Figure 6 It is a schematic functional module diagram of an encryption-enhanced generation system applying a large language model as shown in some embodiments of this specification. Detailed implementation manners

[0022] To more clearly illustrate the technical solutions of the embodiments of this specification, the embodiments will be introduced in detail below with reference to the accompanying drawings. Obviously, the content described below is some examples or embodiments of this specification. For those of ordinary skill in the art, without creative efforts, the technical solutions or means disclosed in this specification can also be applied to other scenarios based on this technical content.

[0023] It should be understood that the "system", "device", "unit" and / or "module" used in this specification is a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the above words can be replaced by other expressions.

[0024] Unless otherwise specified, the technical terms describing components, elements, etc. in this specification do not specifically refer to the singular form, but may also include the plural form. Generally speaking, terms such as "including" and "comprising" only indicate the inclusion of the clearly identified steps, elements or components, and these steps, elements and components do not constitute an exclusive list. For example, the method or device described may also include other steps or components.

[0025] Flowcharts are used in this specification to illustrate the operation steps performed by the devices or systems of the relevant embodiments. However, unless otherwise specified, the order in which these steps are described should not be construed as a limitation on the order of step execution. Those of ordinary skill in the art can adjust the order of execution of these steps according to the knowledge and information conveyed by the embodiments of this specification. The above adjustments include, but are not limited to, the reversal of the sequence, the combination of multiple steps, and the splitting of a certain step.

[0026] Figure 1 It is a schematic structural diagram of a requirement feedback system applying a large language model as shown in some embodiments of this specification. As Figure 1 shown, the requirement feedback system 100 applying a large language model may include a processing device 110, a network 120, a storage medium 130, and a terminal device 140. The processing device 110 and the terminal device 140 can perform data transmission through the network 120. In some embodiments, as Figure 1The demand feedback system 100 applying large language models shown can be applied to fields such as clinical research, medical assistance, content generation, health maintenance, online self-diagnosis, and post-diagnosis follow-up, etc., which are not limited here.

[0027] Among them, the processing device 110 can be a computer device with relatively high computing performance, used to process the demand information input by the user through the large language model and generate the feedback result required by the user. In some embodiments, the processing device 110 can be a single computer device or a computing cluster composed of multiple computer devices, so as to provide more powerful computing power and more efficient response to the demand information input by the user. In some embodiments, the processing device 110 can be a server; specifically, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Distribute Network), and big data and artificial intelligence platforms.

[0028] The network 120 can be any form of wired or wireless network, or any combination thereof. Only as an example, the network 120 can be one or more combinations of a wired network, a fiber optic network, a telecommunications network, an internal network, the Internet, a local area network (LAN), a wide area network (WAN), a wireless local area network (WLAN), a metropolitan area network (MAN), a public switched telephone network (PSTN), a Bluetooth network, etc. The network 120 can have multiple access points, and the processing device 110 and the terminal device 140 can access the network 120 through the access points.

[0029] The storage medium 130 can be used to store data and / or instructions related to the demand feedback system 100 applying large language models. In some embodiments, the storage medium 130 can store data and / or information obtained from the processing device 110, the terminal device 140, etc. As an example, the storage medium 130 can store material content information for generating feedback results, a trained large language model, etc. In some embodiments, the storage medium 130 can include one or more storage components, and each storage component can be an independent device or a part of other devices. In some embodiments, the storage medium 130 can be set in the processing device 110. In some embodiments, the storage medium 130 can include a random access memory (RAM), a read-only memory (ROM), a mass storage device, a removable storage device, a volatile read-write memory, etc. or any combination thereof. Exemplarily, the mass storage device can include a magnetic disk, an optical disk, a solid state disk, etc. In some embodiments, the storage medium 130 can be implemented on a cloud platform.

[0030] The terminal device 140 may include, but is not limited to, a desktop computer, a smart phone, a laptop computer, a VR (Virtual Reality) device, and a tablet computer. The terminal device 140 may be a document recommendation application client, a browser client or an instant messaging client carrying a document search program and a file recommendation program. The user may install and run the application corresponding to the demand feedback system using the large language model on the terminal device 140, display the feedback results through the terminal device, input the demand information through the human-machine interface device, etc., wherein the human-machine interface device may be a part of the terminal device 140, or be separated from the terminal device 140, but have a signal connection with each other. As an example, the terminal device 140 may have a signal connection with the processing device 110 through the network 120, and the user may send the demand information to the processing device 110, and obtain the feedback results corresponding to the demand information from the processing device 110.

[0031] It should be noted that Figure 1 The demand feedback system 100 using a large language model is only an example. The demand feedback system and scenario using a large language model described in the embodiments of this specification are intended to more clearly illustrate the technical solution of the embodiments of this specification, and do not constitute a limitation on the technical solution provided by the embodiments of this specification. For example, in the application scenario of a locally deployed large language model, Figure 1 The processing device 110, network 120 and / or storage medium 130 shown can be omitted. It is known to those skilled in the art that with the emergence of new application scenarios of large language models, the technical solutions provided in the embodiments of this specification are also applicable to similar technical problems. For example, in some application scenarios, the processing device 110, storage medium 130 and terminal device 140 can be integrated into one, and the network 120 can be omitted. For example, in some application scenarios, the processing device 110 and the terminal device 140 can be integrated into one.

[0032] In some embodiments, considering the large amount of professional knowledge and academic terms in the field of professional research, in order to avoid the problem of hallucination in the processing of professional knowledge in the field of professional research by large language models, that is, the content generated by large language models seems reasonable on the surface, but the actual content is fictional or inaccurate information, an enhanced generation knowledge base containing professional knowledge information can be connected to the large language model to overcome this problem. Specifically, the demand information input by the user can be semantically matched with the professional knowledge information stored in the enhanced generation knowledge base to obtain the professional knowledge information associated with the demand information. Then, the obtained professional knowledge information and the demand information are used as prompts (Prompts) to be input into the large language model, so as to provide richer reference information for the large language model and help the large language model generate more accurate feedback results. However, the professional knowledge information stored in the enhanced generation knowledge base is prone to the problem of leakage of professional knowledge information during the process of assisting the large language model in content generation; in addition, the professional knowledge information contained in the enhanced generation knowledge base may also contain classified content that needs to be kept confidential, which is also prone to being illegally obtained or leaked during the process of assisting the large language model in content generation. Malicious attackers can easily illegally obtain the classified content stored in the enhanced generation knowledge base during the semantic matching process between the demand information and the professional knowledge information stored in the enhanced generation knowledge base, which has a significant impact on the data storage security of the enhanced generation knowledge base.

[0033] To overcome the above problems, this specification provides an encrypted enhanced generation method, system and program product for applying large language models. Specifically, Figure 2 is an exemplary flowchart of an encrypted enhanced generation method for applying large language models shown in some embodiments of this specification. In some embodiments, Figure 2 the process 200 shown can be executed by the processing device 110. In some embodiments, the process 200 can be implemented by an encrypted enhanced generation system 600 for applying large language models deployed on the processing device 110. In some embodiments, as Figure 2 shown, the process 200 may include the following steps.

[0034] Step 220: Based on the demand information input by the user, obtain the first encrypted vector corresponding to the demand information through a preset encryption model. In some embodiments, step 220 can be implemented by a demand encryption module 610.

[0035] In some embodiments, the required information input by the user can be the content expected to generate a response through a large language model, and the large language model can process the input required information and generate the required feedback result for the user. For example, when the user needs to continuously follow the latest diagnosis and treatment plans for a certain type of specific disease (such as gastric malignant tumor, etc.) in the field of clinical research, the user can input "Check the latest diagnosis and treatment plans for gastric malignant tumor" as the required information. The large language model can, through understanding the above required information, feedback to the user the latest diagnosis and treatment plans for diseases related to gastric malignant tumor and the corresponding treatment results for each diagnosis and treatment plan. In some embodiments, the large language model (Large Language Model, LLM) can be a type of natural language processing model based on deep learning that has been trained with a large amount of text data and is capable of processing and generating natural language text.

[0036] In some embodiments, the user can freely input the required information according to their own needs. The specific content of the required information can include text content, voice content, image content, etc., which is not limited herein. In some embodiments, the user can input the required information according to the input methods supported by the processing device or terminal device deployed according to Process 200. For example, a text input of the required information can be performed using an external human-machine interface device such as a mouse or keyboard, or a voice input of the required information can be performed using an external human-machine interface device such as a microphone or microphone, which is not limited herein.

[0037] In some embodiments, for the required information input by the user, it can be encrypted through a preset encryption model to generate a corresponding first encrypted vector. It can be understood that considering that the required information input by the user and the professional knowledge information stored in the enhanced generation knowledge base may involve classified content that needs to be kept confidential. Taking the field of clinical research as an example, the required information input by the user may contain classified content such as the diagnosis and treatment plans for certain specific diseases and their feasibility evaluations; at the same time, in order to avoid the problem of processing hallucinations occurring during the process of the large language model processing the required information, the professional knowledge information in the enhanced generation knowledge base associated with the field of clinical research may also contain classified content such as clinical trial plans and clinical guidelines for certain specific diseases. In some embodiments, in order to prevent the disclosure and / or leakage of classified content during the process of using the large language model to generate feedback results for the required information, an encryption algorithm can be selected to encrypt and protect the required information and the professional knowledge information.

[0038] In some embodiments, symmetric algorithms including the AES (Advanced Encryption Standard) algorithm and / or asymmetric algorithms can be used to implement the encryption of demand information and professional knowledge information. In some embodiments, in order to further avoid the increased inference latency of the demand feedback system that uses large language models caused by the high computational complexity of traditional encryption and decryption algorithms, a vectorized coding method can be used to implement the encryption of demand information and / or professional knowledge information. In some embodiments, the Transformer architecture can be used to implement the vectorized coding process for demand information and / or professional knowledge information. Since the Transformer architecture can utilize matrix operations for acceleration and has strong parallel computing capabilities, it can significantly improve the encryption computing efficiency for demand information and / or professional knowledge information. In some embodiments, considering that traditional encryption and decryption algorithms, such as the RSA (Rivest-Shamir-Adleman) algorithm, AES algorithm, etc., may face the threat of quantum computing attacks. For example, quantum computing can quickly solve the large number factorization problem through the Shor algorithm, thereby achieving the deciphering of traditional encryption and decryption algorithms. By adopting the vectorized coding method, which does not rely on traditional mathematical algorithms in implementation, it can have a certain resistance to quantum computing attacks and can further enhance the encryption security guarantee for professional knowledge information. It can be understood that by vectorizing and coding the demand information input by the user, while ensuring the transmission security of the demand information, it is convenient for subsequent retrieval and matching with the encrypted knowledge vectors corresponding to the professional knowledge information stored in the enhanced generation knowledge base associated with the large language model. The specific settings and training process of the preset encryption model will be described in detail later and will not be elaborated here.

[0039] In some embodiments, the demand information input by the user may include information with text as the content carrier. The preset encryption model can generate a corresponding first encryption vector by performing vectorized coding processing on the text content. In some embodiments, when the demand information input by the user contains other content carriers in addition to text, it can be converted into text and then vectorized and coded. For example, for the demand information input by the user through voice, the content of the voice input can be first recognized and converted into the corresponding text content, and then the preset encryption model can perform vectorized coding on the converted text content. In some embodiments, when the input demand information contains two or more different forms of content carriers at the same time, the preset encryption model can be independently used for vectorized coding processing according to the different content carriers. For example, when the demand information input by the user contains both text content and picture content at the same time, the text content and picture content can be respectively vectorized and coded using the preset encryption model, which is not limited here.

[0040] Step 230 : Match the first encrypted vector with the encrypted knowledge vector in terms of similarity, so as to determine the encrypted knowledge vector that matches the first encrypted vector as the second encrypted vector. In some embodiments, step 230 may be implemented by the knowledge matching module 620 .

[0041] In some embodiments, the encrypted knowledge vector is the encrypted information obtained by vectorizing the professional knowledge information required by the large language model. Taking the clinical research field as an example, when the professional knowledge information is a clinical guideline, a single clinical guideline can be split into multiple text segments according to paragraphs or sections, and each text segment is vectorized to obtain the corresponding encrypted knowledge vector. In some embodiments, the encrypted knowledge vector can be obtained by vectorizing the professional knowledge information stored in the enhanced generated knowledge base associated with the large language model based on the encryption model used in the aforementioned step 220. It can be understood that when the encrypted knowledge vector and the first encrypted vector are obtained using the same encryption model based on vectorization coding, if the encrypted knowledge vector and the first encrypted vector have a high degree of similarity, then the professional knowledge information corresponding to the encrypted knowledge vector and the demand information corresponding to the first encrypted vector also have a high degree of similarity, and the professional knowledge information related to the demand information input by the user can be determined by matching the first knowledge vector with the encrypted knowledge vector for similarity.

[0042] In some embodiments, in the process of matching the first encrypted vector with the encrypted knowledge vector for similarity, cosine similarity can be used as a matching standard for similarity, wherein cosine similarity is an indicator for measuring the similarity between two vectors, and the similarity between the two vectors is determined by calculating the cosine value of the angle between the two vectors, and the value range of cosine similarity is in the interval of [-1, 1]. In some embodiments, the cosine similarity between the first encrypted vector and all encrypted knowledge vectors can be obtained first, and then the encrypted knowledge vector whose cosine similarity belongs to the preset range can be determined as the second encrypted vector. For example, the encrypted knowledge vector whose cosine similarity is greater than 0.8 or greater than 0.9 can be determined as the second encrypted vector. Those skilled in the art can define a suitable preset range according to actual needs, which is not limited here. In some embodiments, the encrypted knowledge vectors can also be sorted in order from high to low according to the cosine similarity, and several encrypted knowledge vectors with the highest sorting can be determined as the second encrypted vector. In some embodiments, the number of second encrypted vectors that are similar to and match the first encrypted vector may be one or more. When the number of second encrypted vectors is multiple, the professional knowledge information and demand information corresponding to the multiple second encrypted vectors may be combined together as input information of a large language model, which is not limited here.

[0043] In some embodiments, considering that the number of encrypted knowledge vectors corresponding to the expertise information is large, the IVF (Inverted File Index) algorithm can be used to improve the retrieval matching efficiency between the first encrypted vector and the encrypted knowledge vectors. Specifically, the encrypted knowledge vectors can be divided into multiple clusters by a clustering algorithm, each cluster contains similar encrypted knowledge vectors, and an inverted index is created for each cluster to record the clustering clusters to which the encrypted knowledge vectors belong; during the retrieval matching process, the similarity between the first encrypted vector and the central vectors of each cluster can be queried, and several clusters with higher similarity can be selected for further in-depth retrieval matching to optimize the retrieval matching process between the first encrypted vector and the encrypted knowledge vectors. In some embodiments, the HNSW (Hierarchical Navigable Small World) algorithm can be used to improve the retrieval matching efficiency between the first encrypted vector and the encrypted knowledge vectors. Specifically, a multi-level graph structure can be constructed based on the encrypted knowledge vectors, where each level contains a different number of nodes, the bottom level contains the data points corresponding to all encrypted knowledge vectors, and the upper level is a sparse graph that helps with fast navigation, which can also optimize the retrieval matching process between the first encrypted vector and the encrypted knowledge vectors. Those skilled in the art can also select other suitable vector retrieval matching algorithms according to actual needs, which are not limited herein.

[0044] In some embodiments, the encrypted knowledge vectors can be stored independently outside the enhanced generation knowledge base associated with the large language model, that is, there is physical isolation between the storage location of the encrypted knowledge vectors and the enhanced generation knowledge base; and in the entire process 200 of implementing the encryption enhancement generation method, the demand information input by the user and / or other relevant data streams do not involve the enhanced generation knowledge base storing the expertise information during transmission and processing, that is, the enhanced generation knowledge base and the plaintext-form expertise information stored therein do not participate in the implementation process of process 200. In some embodiments, malicious attackers cannot directly access the enhanced generation knowledge base or obtain the expertise information stored in the enhanced generation knowledge base during the implementation of the above process 200, which can ensure that the expertise information stored in the enhanced generation knowledge base is not illegally obtained. The specific settings and training process of the decryption model will be described in detail later and will not be elaborated herein.

[0045] Step 240: Based on the second encrypted vector and the user key corresponding to the user, obtain the expertise information corresponding to the second encrypted vector through the decryption model corresponding to the user key. In some embodiments, step 240 can be implemented by the knowledge decryption module 630.

[0046] In some embodiments, the user key is a code information for decrypting the second encryption vector obtained by matching in the foregoing process steps, so that the second encryption vector can be restored to the corresponding professional knowledge information by the corresponding decryption model for use by the large language model. In some embodiments, there is a one-to-one binding relationship between the user key and the user of the input demand information, that is, each user who is allowed to use the demand feedback system 100 that applies the large language model as shown in Figure 1 has a corresponding user key, and there are differences between the user keys corresponding to different users. In some embodiments, during the process of a user registering to request the use of the demand feedback system 100 that applies the large language model as shown in Figure 1 , a corresponding user key can be assigned to the user at the initial registration stage. In some embodiments, for the user, the assigned user key can be perceptible to the user. For example, when the user needs to access the demand feedback system 100, the user can implement user identity authentication and verification of the user access operation by inputting the user key, and the demand feedback system 100 can synchronously obtain the user key. In some embodiments, for the user, the assigned user key may not be perceptible to the user. For example, the demand feedback system 100 may store a mapping table including the correspondence between users and user keys. When the user needs to access the demand feedback system 100, the demand feedback system 100 can perform identity authentication based on the login information provided by the user and automatically obtain the user key corresponding to the user. In some embodiments, the user key can also be input by the user when inputting the demand information. For example, the user can input the user key additionally after completing the input of all the demand information, which is not limited herein. In some embodiments, different basic algorithms can be used to generate and obtain the user key, which is not limited herein.

[0047] In some embodiments, there is a one-to-one binding relationship between the user key and the decryption model, that is, each user who is allowed to use the demand feedback system 100 that applies the large language model as shown in Figure 1Each user of the demand feedback system 100 applying the large language model corresponds to a decryption model, and there are differences between the decryption models corresponding to different users. In some embodiments, if a malicious attacker attempts to obtain the professional knowledge information to be encrypted corresponding to the second encryption vector by stealing the user key, even if the malicious attacker successfully steals the user key, only the decryption model corresponding to the stolen user key needs to be invalidated, which can prevent the malicious attacker from obtaining the professional knowledge information corresponding to the second encryption vector. In some embodiments, when it is determined that the first user key is stolen, the corresponding first decryption model can be obtained according to the stolen first user key, and the first decryption model can be invalidated to prevent a third party from using the stolen first user key to decrypt the encrypted knowledge vector so as to obtain the professional knowledge information. The specific settings and training process of the decryption model will be specifically described later and will not be elaborated here.

[0048] Step 250: Based on the professional knowledge information and the demand information, generate a feedback result corresponding to the demand information through the large language model. In some embodiments, step 250 can be implemented by the demand feedback module 640. It can be understood that introducing the professional knowledge information can help the large language model avoid the problem of processing hallucinations in the process of generating the feedback result corresponding to the demand information; at the same time, the professional knowledge information is in an encrypted state of vectorized encoding before being applied by the large language model, which can ensure the privacy and security of the professional knowledge information in the process of applying the large language model for demand information processing and feedback result generation.

[0049] In some embodiments, such as Figure 2The process 200 shown may further include step 210: verifying the legality of the large language model generation request input by the user based on a preset license key, and turning to step 220 when the legality verification passes. It can be understood that before vectorizing and encoding the demand information input by the user, it is possible to first verify whether the user has the qualification to access the large language model and provide demand information to the large language model. In some embodiments, the large language model generation request in step 210 may include a user key and demand information, and the large language model may use its own preset license key to verify the legality of the user information, such as the user key, included in the large language model generation request. In some embodiments, it is also possible to match the decryption model according to the user key included in the large language model generation request. If there is a decryption model that matches the user key, it indicates that the legality verification passes; if there is no decryption model that matches the user key, it indicates that the legality verification fails. Those skilled in the art can select a suitable legality verification method, such as symmetric encryption and decryption verification, to implement the legality verification, which is not limited herein. In some embodiments, for the large language model generation request that passes the legality verification, the large language model generation request can also be recorded in the form of a log (log). Specifically, the log may include the initiation timestamp corresponding to the large language model generation request, the initiation user information corresponding to the large language model generation request, the demand information corresponding to the large language model generation request, etc., which is not limited herein, and the initiation and operation of the large language model generation request can be audited and traced through the log.

[0050] Based on the relevant descriptions of the foregoing embodiments, it is possible to achieve the security guarantee of the professional knowledge information required by the large language model during the content generation process. The following will further elaborate and explain the specific settings and training processes of the encryption model and decryption model involved in the above process 200 in combination with embodiments.

[0051] Figure 3 is an exemplary flowchart of a preset process of an encryption model according to some embodiments of this specification. In some embodiments, as Figure 3 shown, the process 300 may include the following steps.

[0052] Step 310: Perform full-parameter adjustment training on a preset natural language processing model to make the input data distribution of the natural language processing model consistent with the output data distribution.

[0053] It can be understood that for the encryption model and decryption model involved in the above process 200, the professional knowledge information can be first converted into high-dimensional vectors, and then the corresponding professional knowledge information can be restored from the high-dimensional vectors. The entire training process can be regarded as a complete self-supervised learning training process. The corresponding encryption model and decryption model can form a complete natural language processing (NLP, Natural Language Processing) model. Among them, the encryption model can be the part before the vector representation stage (Embedding) in the natural language processing model, and the decryption model is the part after the vector representation stage in the natural language processing model. In some embodiments, the preset natural language processing model can be trained by adjusting its parameters, and the part before the vector representation stage in the natural language processing model after the first parameter adjustment training can be used as the preset encryption model, and the part after the vector representation stage in the natural language processing model after the second parameter adjustment training can be used as the preset decryption model. In some embodiments, the preset natural language processing model can be the open-source Qwen large model, which is based on the Transformer architecture and supports pre-training of large-scale text data and full-parameter adjustment for specific tasks. Those skilled in the art can also select other suitable natural language processing models, which are not limited here.

[0054] In some embodiments, the training process of the encryption model involves full-parameter adjustment training of the natural language processing model, taking the input training data set as an example, the output result of the natural language processing model can be expressed as , where is the part before the preset vector representation stage of the natural language processing model, that is, the preset encryption model part with trainable parameters; is the part after the preset vector representation stage of the natural language processing model, that is, the preset decryption model part with trainable parameters; is the output result of the natural language processing model. It should be noted that since the decryption model is obtained from the second training, the parameters of the decryption model part obtained in this training are not of practical use. In order to ensure that the encryption model and the decryption model can realize the vectorized encoding and restoration of professional knowledge information, it is required that the input data distribution after full-parameter adjustment training is consistent with the output data distribution, that is, the distribution of is consistent with the distribution of . In some embodiments, in order to achieve the distribution of being consistent with the distribution of , the optimization target can be set as the combination of the loss function of the natural language processing model and the loss function of the encryption model in the full-parameter adjustment training. For a single sample The loss function of the natural language processing model can be expressed as , where is the sample output, the r-th token of the prediction probability at is the sample the input content composed of the first r tokens in is the length of that is, the number of tokens in ; at the same time, in order to make the output of the encryption model part quantitatively measure the semantic similarity in the feature space, it is necessary to impose a semantic embedding constraint on The loss function of the encryption model for a single sample can be expressed as which represents the vector representation output of the encryption model for the sample , represents the positive sample of the sample in a training batch (batch) (where the positive sample is a sample formed by randomly occluding or augmenting some tokens of the sample ), represents the negative sample of the sample in a training batch (batch) (where the negative sample is the other samples in a training batch except the sample ), , used to represent the cosine similarity between the vectors and , is the temperature hyperparameter that controls the distribution smoothness, is the total number of samples in a training batch (batch). Then the total loss can be expressed as , where is used to control and Hyperparameters for target balance. In some embodiments, any reasonable setting of the training loss function falls within the scope of protection of this specification. In some embodiments, the gradual unfreezing algorithm and the discriminative fine-tuning algorithm can be adopted in the full-parameter adjustment training. Specifically, in the first 5 training epochs, by fixing the parameters of the first 10 layers of the natural language model and only training the subsequent layers, the convergence speed of the model can be accelerated; in the subsequent 10 training epochs, the fixed parameters of the first 10 layers of the natural language model are unfrozen layer by layer to ensure the stability of the model; during the unfreezing process, the learning rate of each layer decreases in a certain proportion, usually starting from the top (i.e., the layer farthest from the input) to the bottom (i.e., the layer closest to the input) of the model. As an example, the learning rate of each layer is reduced by 2.6 times compared to the previous layer. In some embodiments, the Qwen large model can select the pre-trained open-source large model Qwen2.5-0.5B. In some embodiments, the training data participating in the full-parameter adjustment training can include the professional knowledge content in the enhanced knowledge database, the relevant knowledge content in the related technical field retrieved through public search engines such as Wikipedia, and the Chinese translations of the relevant knowledge content, etc., which are not limited here.

[0055] In some embodiments, since the high-dimensional vectors obtained by the encrypted model need to have good semantic representations for retrieval matching between the first encrypted vector and the encrypted knowledge vector, the training results of the encrypted model can be evaluated using the Relatedness standard evaluation, that is, evaluating the similarity degree between two texts after vectorized encoding by the encrypted model. The similarity degree between two similar texts after vectorized encoding needs to be high, and the similarity degree between two unrelated texts after vectorized encoding needs to be low. Specifically, the cosine similarity can be used to judge the similarity degree, which is not limited here.

[0056] Step 320: Determine the part before the vector representation stage in the natural language processing model after full-parameter adjustment training as the encrypted model. It can be understood that the part before the vector representation stage in the natural language processing model after full-parameter adjustment training is determined as the encrypted model and fixed, so as to directly use the part before the vector representation stage in the natural language processing model during the subsequent training process of the decryption model.

[0057] Figure 4 It is an exemplary flowchart of a preset process of a decryption model according to some embodiments of this specification. In some embodiments, such as Figure 4 The process 400 shown is executed after the process 300 shown in Figure 3 In some embodiments, such as Figure 4As shown, process 400 may include the following steps.

[0058] Step 410: Introduce a randomly initialized user feature vector into the part after the vector representation stage in the natural language processing model after full-parameter adjustment, and perform local parameter adjustment training to make the input data distribution of the natural language processing model consistent with the output data distribution.

[0059] In some embodiments, the training process of the decryption model involves local parameter adjustment training of the natural language processing model. Taking the input training data set as an example, the output result of the natural language processing model can be expressed as , where is the encrypted model part with fixed parameters obtained after the full-parameter adjustment training of the aforementioned process 300; is the part after the preset vector representation stage of the natural language processing model, that is, the preset decryption model part with trainable parameters; is the output result of the natural language processing model; is the randomly initialized trainable user feature vector introduced to ensure user diversity and vector randomness. In some embodiments, in order to achieve the distribution of to be consistent with the distribution of , the optimization target can be set as the loss function of the natural language processing model in the local parameter adjustment training, that is, , where is the predicted probability of the r-th token of the sample output in the sample formed by the first r tokens in , R is the length of (that is, the number of tokens in ), and is the total number of samples in a training batch (batch). In some embodiments, any reasonable setting of the training loss function falls within the scope of protection of this specification. In some embodiments, can also be a series of user feature vectors preset according to rules, which are not limited here. In some embodiments, the user feature vector can be added before the input of any layer in the decryption model , and the decryption model formed after local parameter fine-tuning training before adding the user feature vector to any layer falls within the scope of protection of this specification. In some more extensive examples, the decryption process may not be limited to the formula , for example, if the user chooses to fine-tune the decryption model part in the LORA manner, at this time, the output of the th layer of the fine-tuned decryption model is , where and are the parameters of the th linear layer of the decryption model obtained after the first full-parameter fine-tuning training in the aforementioned process 300, and are the low-rank update terms of the th layer of the decryption model obtained after the second LORA fine-tuning training, , at this time, the low-rank update terms of any layer or any combination of layers can be used as the user feature vector , S is the number of linear layers of the decryption model obtained after the second LORA fine-tuning training . The decryption models formed after the above LORA fine-tuning training are all within the protection scope of this specification. In order to ensure that the decryption model and the decryption model can realize the vectorized encoding and restoration of professional knowledge information, it is required that the input data distribution after local parameter adjustment is consistent with the output data distribution, that is, 's distribution is consistent with 's distribution.

[0060] Step 420: Determine the part after the vector representation stage in the natural language processing model after local parameter adjustment training as the decryption model, where the decryption model corresponds one-to-one with the user feature vector, and use the user feature vector as the user key. It can be understood that for the decryption model obtained after local parameter adjustment training, there is a one-to-one correspondence between it and the user feature vector (i.e., the user key), that is, only when the user key is correct can the decryption model restore the encrypted knowledge vector to the corresponding professional knowledge information, effectively ensuring the data security of professional knowledge information in the application process of the large language model.

[0061] One or more embodiments of this specification also provide an encryption-enhanced generation system applying a large language model. Figure 5 is a data flow diagram of an encryption-enhanced generation system applying a large language model shown according to some embodiments of this specification. In some embodiments, as Figure 5 shown, the encryption-enhanced generation system 500 applying a large language model may include a first server 510 and a second server 520.

[0062] In some embodiments, the first server 510 may be used to obtain a first encrypted vector corresponding to the requirement information based on the requirement information input by the user through a preset encryption model. In some embodiments, the first server 510 may also be used to perform a similarity degree matching between the first encrypted vector and the encrypted knowledge vector, so as to determine the encrypted knowledge vector that matches the first encrypted vector as the second encrypted vector and output it. In some embodiments, the first server 510 includes a storage space for storing encrypted knowledge vectors. In some embodiments, the first server 510 may be a server provided by a service provider for providing professional knowledge information.

[0063] In some embodiments, the second server 520 is a server independent of the first server 510, and is used to obtain professional knowledge information corresponding to the second encrypted vector based on the second encrypted vector and the user key corresponding to the user through a decryption model corresponding to the user key. In some embodiments, the large language model is carried on the second server 520, and can generate a corresponding feedback result based on the professional knowledge information and the requirement information and feedback it to the user. In some embodiments, the second server may be a third-party server for providing large language model requirement feedback services.

[0064] In some embodiments, as Figure 5 shown, the encryption-enhanced generation system 500 applying the large language model may further include an enhanced generation knowledge base 530 for storing professional knowledge information. In some embodiments, the first server 510 includes an encrypted knowledge vector library, and is used to obtain and store encrypted knowledge vectors based on the professional knowledge information stored in the enhanced generation knowledge base 530 through the encryption model carried by the first server. In some embodiments, the encrypted knowledge vector library and the enhanced generation knowledge base 530 are separated from each other and independently deployed.

[0065] Some embodiments of this specification also provide an encryption-enhanced generation system applying a large language model. Figure 6 It is a schematic diagram of the functional modules of an encryption-enhanced generation system applying a large language model shown according to some embodiments of this specification. In some embodiments, as Figure 6 shown, the encryption-enhanced generation system 600 applying the large language model may include a requirement encryption module 610, a knowledge matching module 620, a knowledge decryption module 630, and a requirement feedback module 640. In some embodiments, each module of the encryption-enhanced generation system 600 applying the large language model may be deployed in the first server 510 and / or the second server 520. In some embodiments, each module of the encryption-enhanced generation system 600 applying the large language model may be configured in the processing device 110.

[0066] In some embodiments, the requirement encryption module 610 can perform vectorized encryption processing on the requirement information input by the user to obtain a corresponding encrypted vector, which facilitates subsequent retrieval and matching with the encrypted knowledge vector while ensuring the transmission security of the requirement information. In some embodiments, the requirement encryption module 610 can be used to obtain a first encrypted vector corresponding to the requirement information based on the requirement information input by the user through a preset encryption model. In some embodiments, the requirement encryption module 610 can be deployed in the first server 510.

[0067] In some embodiments, the knowledge matching module 620 can match the encrypted vector corresponding to the requirement information input by the user with the encrypted knowledge vector corresponding to the professional knowledge information stored in the enhanced generation knowledge base associated with the large language model to obtain the encrypted knowledge vector corresponding to the professional knowledge information associated with the requirement information. In some embodiments, the knowledge matching module 620 can be used to perform a similarity matching between the first encrypted vector and the encrypted knowledge vector to determine the encrypted knowledge vector that matches the first encrypted vector as the second encrypted vector. In some embodiments, the knowledge matching module 620 can be deployed in the first server 510.

[0068] In some embodiments, the knowledge decryption module 630 can be used to decrypt the second encrypted vector obtained by the knowledge matching module 620 that matches the requirement information to obtain the professional knowledge information corresponding to the second encrypted vector, and this professional knowledge information can be used by the large language model in the subsequent feedback result generation process. In some embodiments, the knowledge decryption module 630 can obtain the professional knowledge information corresponding to the second encrypted vector based on the second encrypted vector and the user key corresponding to the user through the decryption model corresponding to the user key. In some embodiments, the knowledge decryption module 630 can be deployed in the second server 520.

[0069] In some embodiments, the requirement feedback module 640 can generate the feedback result required by the user through the large language model according to the requirement information input by the user. In some embodiments, in order to avoid the problem of processing hallucinations occurring in the large language model during the processing due to a large amount of professional knowledge and / or academic terms in the requirement information input by the user, the requirement feedback module 640 can also use the professional knowledge information associated with the requirement information provided by the knowledge decryption module 630 to assist in generating the feedback result during the process of the large language model processing the requirement information. In some embodiments, the requirement feedback module 640 can be used to generate the feedback result corresponding to the requirement information through the large language model based on the requirement information and the professional knowledge information matching the requirement information. In some embodiments, the requirement feedback module 640 can be deployed in the second server 520.

[0070] For more content about each module, please refer toFigures 2 to 5 The relevant descriptions are not elaborated here. It should be understood that Figure 6 The system and its modules shown can be implemented in various ways. For example, in some embodiments, the system and its modules can be implemented through hardware, software, or a combination of software and hardware. Among them, the hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. Those skilled in the art can understand that the above methods and systems can be implemented using computer-executable instructions and / or control codes included in a processor. For example, such codes are provided in carrier media such as magnetic disks, CDs, or DVD-ROMs, or in the memories of programmable devices. The system and its modules in this specification can be implemented not only by hardware circuits such as very large scale integrated circuits or gate arrays, semiconductors such as logic chips and transistors, or programmable hardware devices such as field programmable gate arrays and programmable logic devices, but also by software executed by various types of processors, or by a combination of the above hardware circuits and software (e.g., firmware).

[0071] It should be noted that the above description of the system and its modules is only for convenience of description and does not limit this specification within the scope of the examples given. It can be understood that for those skilled in the art, after understanding the principle of the system, they may, without departing from this principle, arbitrarily combine the various modules to form a subsystem connected to other modules. Or split some modules to obtain more modules or multiple units under that module. Such deformations are all within the scope disclosed in this specification.

[0072] Some embodiments of this specification also provide an encryption-enhanced generation device applying a large language model, which includes a processor and a storage medium. Computer program instructions are stored in the storage medium, and the processor is used to execute at least part of the computer program instructions to implement the encryption-enhanced generation method applying a large language model provided in the foregoing embodiments of this specification.

[0073] Some embodiments of this specification also provide a computer program product, including computer instructions or a computer program. When at least part of the computer instructions or the computer program is executed by a processor, it can implement the encryption-enhanced generation method applying a large language model provided in the foregoing embodiments of this specification.

[0074] In some embodiments, the above-mentioned processor may be a combination of one or more of the following processors: central processing unit (CPU), application-specific integrated circuit (ASIC), application-specific instruction set processor (ASIP), graphics processing unit (GPU), physics processing unit (PPU), digital signal processor (DSP), field-programmable gate array (FPGA), programmable logic device (PLD), programmable logic controller (PLC), reduced instruction set computer (RISC), microprocessor, etc. In some embodiments, the processor may be the processing device 110.

[0075] The beneficial effects that the embodiments of this specification may bring include, but are not limited to: (1) By introducing professional knowledge information, it can help the large language model avoid the problem of processing hallucinations during the process of generating the feedback result corresponding to the requirement information; at the same time, the professional knowledge information is in an encrypted state of vector quantization encoding before being applied by the large language model, which can ensure the privacy and security of the professional knowledge information during the process of using the large language model for requirement information processing and feedback result generation. (2) Using the method of vector quantization encoding to implement the encryption of requirement information and / or professional knowledge information can significantly improve the encryption / decryption calculation efficiency of requirement information and / or professional knowledge information. (3) The encrypted knowledge vector corresponding to the professional knowledge information is physically isolated from the storage location of the professional knowledge information, which can avoid directly accessing the storage location of the professional knowledge information during the application process of the large language model, so as to improve the storage security of the professional knowledge information. It should be noted that the beneficial effects that may be produced by different embodiments are different. In different embodiments, the beneficial effects that may be produced may be any one or several combinations of the above, or any other beneficial effects that may be obtained.

[0076] The basic concepts have been described above. Obviously, for those skilled in the art, the above detailed disclosure is only an example and does not constitute a limitation to this specification. Although not explicitly stated here, those skilled in the art may make various modifications, improvements, and corrections to this specification. Such modifications, improvements, and corrections are taught in this specification, so such modifications, improvements, and corrections still belong to the spirit and scope of the exemplary embodiments of this specification.

Claims

1. An encryption-enhanced generation method using large language models, characterized in that, The method includes: Based on the requirement information input by the user, obtaining a first encrypted vector corresponding to the requirement information through a preset encryption model; Performing a similarity degree matching between the first encrypted vector and an encrypted knowledge vector to determine the encrypted knowledge vector that matches the first encrypted vector as a second encrypted vector; Based on the second encrypted vector and the user key corresponding to the user, obtaining professional knowledge information corresponding to the second encrypted vector through a decryption model corresponding to the user key; Based on the professional knowledge information and the requirement information, generating a feedback result corresponding to the requirement information through a large language model.

2. The encryption enhancement generation method using a large language model according to claim 1, wherein The encrypted knowledge vector is obtained by performing vectorized encryption processing on the professional knowledge information stored in an enhanced generation knowledge base associated with the large language model through the encryption model; Wherein, the encrypted knowledge vector is stored independently outside the enhanced generation knowledge base.

3. The encryption-enhanced generation method using a large language model according to claim 1, characterized in that The method further includes: Performing a legality verification on a large language model generation request input by the user based on a preset license key, where the large language model generation request includes the user key and the requirement information; In the case where the legality verification passes, obtaining the first encrypted vector corresponding to the requirement information through the encryption model.

4. The encryption-enhanced generation method using a large language model according to claim 1, wherein The performing a similarity degree matching between the first encrypted vector and an encrypted knowledge vector to determine the encrypted knowledge vector that matches the first encrypted vector as a second encrypted vector includes: Obtaining the cosine similarity between the first encrypted vector and the encrypted knowledge vector; Determining the encrypted knowledge vector whose cosine similarity belongs to a preset range as the second encrypted vector.

5. The encryption-enhanced generation method using a large language model according to claim 1, characterized in that, The preset process of the encryption model includes: Performing full-parameter adjustment training on a preset natural language processing model to make the input data distribution and the output data distribution of the natural language processing model consistent; Determining the part before the vector representation stage in the natural language processing model after the full-parameter adjustment training as the encryption model.

6. The encryption-enhanced generation method using a large language model according to claim 5, wherein The preset process of the decryption model includes: Introducing a randomly initialized user feature vector into the part after the vector representation stage in the natural language processing model after the full-parameter adjustment and performing local parameter adjustment training to make the input data distribution and the output data distribution of the natural language processing model consistent; Determining the part after the vector representation stage in the natural language processing model after the local parameter adjustment training as the decryption model, where the decryption model corresponds one-to-one with the user feature vector; and Using the user feature vector as the user key.

7. An encryption-enhanced generation system using a large language model, characterized in that, The system includes: A first server, configured to obtain a first encrypted vector corresponding to the requirement information input by the user through a preset encryption model; and configured to perform a similarity degree matching between the first encrypted vector and an encrypted knowledge vector to determine the encrypted knowledge vector that matches the first encrypted vector as a second encrypted vector and output it; A second server, configured to obtain, based on the second encrypted vector and the user key corresponding to the user, the professional knowledge information corresponding to the second encrypted vector through a decryption model corresponding to the user key; A large language model is deployed on the second server, configured to generate a corresponding feedback result based on the professional knowledge information and the requirement information.

8. The encryption-enhanced generation system using a large language model according to claim 7, characterized in that, The system further includes: An enhanced generation knowledge base, configured to store the professional knowledge information; The first server includes an encrypted knowledge vector library, which is configured to obtain and store the encrypted knowledge vector based on the professional knowledge information stored in the enhanced generation knowledge base through the encryption model; The encrypted knowledge vector library and the enhanced generation knowledge base are separated from each other and independently deployed.

9. An encryption-enhanced generation system applying a large language model, characterized in that, The system includes: A requirement encryption module, configured to obtain, based on the requirement information input by the user, a first encrypted vector corresponding to the requirement information through a preset encryption model; A knowledge matching module, configured to match the similarity between the first encrypted vector and the encrypted knowledge vectors, and determine the encrypted knowledge vector that matches the first encrypted vector as the second encrypted vector; A knowledge decryption module, configured to obtain, based on the second encrypted vector and the user key corresponding to the user, the professional knowledge information corresponding to the second encrypted vector through a decryption model corresponding to the user key; A requirement feedback module, configured to generate a feedback result corresponding to the requirement information through a large language model based on the professional knowledge information and the requirement information.

10. A computer program product, comprising computer instructions or a computer program, characterized in that, When at least part of the computer instructions or the computer program is executed by a processor, it can implement the encrypted enhanced generation method using a large language model as described in any one of claims 1 to 6.

Images