A label calculation method based on hybrid expert LLM

Abstract

The application discloses a label calculation method based on a hybrid expert LLM, and the method comprises the following steps: designing a label calculation method of a hybrid expert LLM system model architecture, and adopting multiple LLM parallel calculation and model integration. The label semantic expansion based on vector similarity, the prior calculation based on LLM+ small sample prior knowledge, the posterior calculation based on LLM+ RAG, the label filtering based on the joint soft decision of multiple LLMs, and the secondary label recall based on knowledge distillation are used to realize accurate calculation of labels. The method is different from a traditional label calculation scheme, greatly improves the accuracy and reliability of label calculation, effectively solves the model illusion problem, and guarantees the accuracy of label calculation.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence information technology, and in particular relates to a label calculation method based on hybrid expert LLM. Background Technology

[0002] Tagging user information, also known as user profiling, involves collecting data from various dimensions such as basic user data, social attributes, consumption habits, and preferences to characterize user or product attributes, uncover potential value information, and thus abstract a complete picture of user information. User profiling is the foundation of big data applications and plays a crucial role in many scenarios such as human resource management, job matching, precision marketing, personalized recommendations, personalized services, and refined operations, serving as the basis for data-driven operations. The calculation of user profile tags, commonly known as "tagging" users, generally falls into three categories: ① statistical tags; ② rule-based tags; and ③ machine learning tags. The first two types of tags are relatively simple to calculate and have relatively low technical complexity. Machine learning tags, however, require AI algorithms to generate tags that predict or intelligently analyze certain user attributes or behaviors. Examples include determining a user's gender based on their behavioral habits, judging their preference for a particular product based on their consumption habits, extracting tags for their work skills from their resume text data, and extracting tags for their interests from their reading and browsing logs.

[0003] This invention primarily addresses the problem of extracting large amounts of tags from unstructured long text. To illustrate the background of this problem and explain the invention's method, this paper will use "extracting job skill tags from resume text data" as an example, which will not be elaborated upon further below. The technical difficulty of this type of problem lies in:

[0004] 1) Low accuracy / recall of label-based computation models. Due to the wide range of label values, such as the potential for tens of thousands of skill labels in a resume, traditional machine learning algorithms typically model label extraction as a multi-label classification or sequence labeling problem. For multi-class problems with an excessively large number of target categories (e.g., tens of thousands), the classification target data is extremely scattered and sparse, making it difficult to guarantee the accuracy / recall of the classification model. Furthermore, the extreme sparsity of the data places high demands on the quality and sample size of the training data. Existing Named Entity Recognition (NER) algorithms, which rely on the annotation of large-scale, high-quality training corpora of domain knowledge, also suffer from low accuracy / recall and difficulty in handling complex semantics.

[0005] 2) Weak semantic understanding. Accurate semantic understanding of tags is extremely complex, with numerous synonyms and ambiguous words. For example, the tags "database" and "data mining" are similar in wording but express completely different job skills—a small difference can lead to a huge error. Similarly, "federated learning" and "machine learning" differ significantly in wording but share considerable semantic similarity. Existing tag calculation methods are still insufficient for natural language processing and understanding, struggling with accurate semantic understanding, ambiguity handling, synonym processing, understanding of vague expressions, and comprehension of spoken language. They also lack comprehensive understanding and reasoning capabilities regarding text semantics, especially long texts, resulting in unsatisfactory core metrics for tag extraction (low accuracy and recall). Conventional models used for text classification, such as Naive Bayes, SVM, Logistic Regression, Random Forest, and XGBoost, generally only capture shallow semantics in text. Even text classification algorithms that integrate pre-trained deep learning models (ELMo, BERT, etc.) have limited practical effectiveness and require extensive secondary training on local data.

[0006] 3) Poor label generation and expansion capabilities. Existing label calculation methods are mainly based on supervised learning to train models, primarily using discriminative models to distinguish (or predict) text or entities, lacking the ability to directly generate labels; at the same time, existing label calculation methods cannot automatically expand to "unknown" (models not trained) label information.

[0007] 4) Insufficient privacy, security, and credibility in tag calculation. During tag calculation, users' raw data (such as resume text data, user behavior data, etc.) often involves user privacy. The original data provider and the calculation party are usually personnel with different roles. The calculation party typically performs calculations based on the raw plaintext provided by the data provider, posing a risk of stealing sensitive user privacy information. Furthermore, the reliability of the calculation process and results lacks a trust mechanism to guarantee its credibility.

[0008] To address the above issues, this invention innovatively solves four core problems in tag calculation for long texts: low accuracy and low recall, weak semantic understanding, poor tag generation and expansion capabilities, and insufficient privacy, security, and credibility in tag calculation. Summary of the Invention

[0009] The purpose of this invention is to address the shortcomings of existing technologies, such as low accuracy and recall, weak semantic understanding, and poor tag generation and expansion capabilities in tag computation from unstructured long texts. It provides a tag computation method based on a hybrid expert LLM (Large Language Model). To address the issues of insufficient privacy, security, and credibility in tag computation, this invention provides a trusted intelligent tag computation device that combines hybrid expert LLM and privacy protection. Furthermore, it addresses two major negative problems arising from the application of LLM in tag computation: model illusion (such as the potential for partially fabricated, fabricated, or unreliable tags) and the inherent lack of credibility within a single LLM (different LLMs have different "biases" in deep semantic understanding of text; even subtle errors in a single LLM can be amplified and trigger erroneous results; a single LLM also suffers from malicious model behavior or unreliable credibility).

[0010] The objective of this invention is achieved through the following technical solution: a tag calculation method based on hybrid expert LLM, comprising:

[0011] Obtain user information and build a corresponding tag library;

[0012] Based on user information and tags in the tag library, prompt1 is formed, which is used to ask questions to M different LLMs in the first hybrid expert model system. The returned results are constrained by a list of tags, and all results are merged to obtain the first tag set.

[0013] The similarity between each tag in the first tag set and the tags in the tag library is calculated, and each tag in the first tag set returns a set of similar tags, thus obtaining the second tag set;

[0014] The tags in the second tag set are converted into vectors for the retrieval query question, and the vector database is searched to obtain the most relevant context information for each tag; the vector database stores user information.

[0015] Insert the context information and its corresponding label into prompt2, and ask N different LLMs in the second hybrid expert model system to determine whether the label is correct or not. If the LLM determines that the label is correct, it is assigned a weight of 1 / N; if it determines that the label is incorrect, it is assigned a weight of 0. If the sum of the weights is greater than or equal to the first preset threshold, the label is retained; if it is less than or equal to the second preset threshold, the label is discarded. If the sum of the weights is between the first and second preset thresholds, the label is marked as a label to be determined.

[0016] If there is a label to be determined, input the label to be determined into the classification model and determine whether the label is correct or not. If it is correct, keep the label; otherwise, discard the label.

[0017] Furthermore, each tag in prompt1 is independently and randomly sampled from the tag library.

[0018] Furthermore, the returned result is constrained to include the following: the tags returned by each LLM are sorted from most important to least important, and the top N tags are returned.

[0019] The returned set of similar tags includes: sorted by similarity from largest to smallest and taking at most the top K tags based on a third preset threshold.

[0020] Furthermore, the user in the user information refers to the subject object to be calculated by the tag, and the user information is the text description data information related to the subject object.

[0021] Furthermore, the classification model is trained as follows:

[0022] Using at least one LLM in the second hybrid expert model system as the teacher model and the classification model as the student model, the classification model is trained by distillation.

[0023] Furthermore, it also includes expanding the tag library, specifically:

[0024] According to the tag list generated by LLM in the first hybrid expert model system, if the frequency of the generated tag is greater than the fourth preset threshold and / or the similarity between the generated tag and the tags in the tag library is greater than the fifth threshold, then the tag is added to the tag library.

[0025] Furthermore, it also includes extending the vector database, specifically:

[0026] The semantic information is extended based on the acquired user information. The extended semantic information is then vectorized and stored in the vector database, or a new vector database is created and stored in the newly created vector database.

[0027] Furthermore, it also includes credit scoring of the LLM in the second hybrid expert model system;

[0028] The reputation score is calculated as follows: R-LLM(i) = sum(LLM(i) == nLLM Output) / cumulative number of decisions; where LLM(i) == nLLM Output means determining whether the single output of the i-th LLM is equal to the final output. If it is, the output is 1, otherwise it is 0.

[0029] If the reputation score is lower than the third preset threshold, the LLM will be removed from the label calculation.

[0030] The present invention also provides a hybrid expert LLM and privacy-preserving trusted smart tag computing device, including a user side and a data computing sandbox;

[0031] The user side includes a local database and a vector database, both of which store encrypted user information;

[0032] The data computation sandbox is used to execute the aforementioned tag computation method based on hybrid expert LLM, obtain the retained tags, encrypt the tags, and output them to the user side for storage; the user information obtained is obtained by retrieving the specified user encrypted information from the local database and decrypting it.

[0033] Furthermore, it also includes regulators, who are responsible for conducting independent and public audits of the computational data, processes, and results.

[0034] Compared with the prior art, the beneficial effects of the embodiments in this specification are:

[0035] (1) Improving the accuracy and recall of label calculation. This method differs from traditional label extraction methods based on machine learning / deep learning. This invention comprehensively applies a design framework combining LLM with a locally self-trained fine-tuning model, and generative and discriminative label calculation. Through multi-path LLM hybrid expert model integration, label semantic expansion based on vector similarity, the introduction of small-sample prior knowledge in the prompt process, and multiple recall based on knowledge distillation, the recall rate of label calculation is comprehensively improved. This invention leverages the inherent powerful semantic understanding and synthesis capabilities of LLM to ensure the accuracy of label calculation. Furthermore, it designs a computational architecture combining prior calculation (initial recall) and posterior calculation (reliability verification), and employs multi-LLM joint decision-making, which further improves the accuracy of label extraction compared to a single LLM. The above algorithm design significantly enhances the accuracy of label calculation.

[0036] (2) Solving the illusion problem in LLM-based label computation. In the first phase of the prompt project (prompt1), this invention uses independent random sampling of the original label library to provide some label value samples as prior knowledge or example hints to the LLM, thereby guiding and constraining the accuracy and standardization of label computation by the LLM, suppressing the possibility of LLM fabricating or forging labels, and reducing the probability of the LLM model illusion problem. At the same time, a single LLM is usually limited by the number of token inputs, and the prior knowledge that can be input at one time (such as tens of thousands of labels in the initial label library) is limited. This method samples the prior knowledge (tens of thousands of labels) in parallel and inputs it into different LLMs as much as possible. From an overall perspective, it can improve the understanding of prior knowledge by multiple LLMs, thereby improving the overall label computation performance. In the second phase of the prompt project (prompt2), this invention introduces RAG technology and soft decision method to verify and screen the reliability of labels, further effectively solving the model illusion problem while ensuring the accuracy of label computation.

[0037] (3) Improved reliability of label computation. This invention innovatively adopts a multi-path parallel hybrid expert LLM computation architecture for label extraction, which can significantly improve the system reliability, computational reliability, and result reliability of label computation without relying on a single LLM (whether it is due to service failure, model illusion, or other problems).

[0038] (4) Achieve automated expansion and updating of the tag library. Due to the inherent text generation capability of LLM technology, this method differs from traditional calculation methods based on classification or sequence labeling problems. This method can automatically calculate and expand some "unknown" and valuable new tags, and after simple classification and verification, it can automatically update the local tag library, thereby achieving automated tag expansion.

[0039] (5) Privacy-preserving trusted computation is achieved. This method uses blockchain technology and a data computing sandbox to ensure the privacy and security of the main algorithm's computation process and the reliability of the computation results. All plaintext information is decrypted and computed only in the sandbox's memory. The original computation data and key process information in the sandbox are written to the blockchain through smart contracts and verified for reliability. The blockchain network is used to ensure that the information is transparent, open, and tamper-proof. Through the above technologies, it is ensured that the tag computation algorithm logic, LLM prompts, and computation input and output data inside the sandbox are not maliciously tampered with, preventing the risk of leaking user privacy data and ensuring the reliability of the results. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a flowchart of a tag calculation method provided in Embodiment 1 of the present invention;

[0042] Figure 2 This is an example (llama3 model) of a prompt language prompt1 provided in Embodiment 1 of the present invention;

[0043] Figure 3 This is an example (qwen2 model) of another prompt language prompt1 provided in Embodiment 1 of the present invention;

[0044] Figure 4 This is a reference template for a prompt language (prompt2) provided in Embodiment 1 of the present invention;

[0045] Figure 5 This is an example (llama3 model) of a prompt language prompt2 provided in Embodiment 1 of the present invention;

[0046] Figure 6 This is an example (qwen2 model) of another prompting language, prompt2, provided in Embodiment 1 of the present invention;

[0047] Figure 7 This is an example (llama3 model) of another prompt language, prompt2, provided in Embodiment 1 of the present invention;

[0048] Figure 8 This is an example (qwen2 model) of another prompt language, prompt2, provided in Embodiment 1 of the present invention;

[0049] Figure 9 This is a schematic diagram of a tag secondary recall process based on knowledge distillation provided in Embodiment 1 of the present invention;

[0050] Figure 10 This is an example (llama3 model) of a prompt language prompt1 provided in Embodiment 2 of the present invention;

[0051] Figure 11 This is an example (qwen2 model) of another prompt language prompt1 provided in Embodiment 2 of the present invention;

[0052] Figure 12This is a reference template for a prompt language (prompt2) provided in Embodiment 2 of the present invention;

[0053] Figure 13 This is an example (llama3 model) of a prompt language prompt2 provided in Embodiment 2 of the present invention;

[0054] Figure 14 This is an example (qwen2 model) of another prompting language, prompt2, provided in Embodiment 2 of the present invention;

[0055] Figure 15 This is an example (llama3 model) of another prompt language, prompt2, provided in Embodiment 2 of the present invention;

[0056] Figure 16 This is an example (qwen2 model) of another prompt language, prompt2, provided in Embodiment 2 of the present invention. Detailed Implementation

[0057] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The singular forms “a,” “the,” and “the” used in this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0058] The present invention will now be described in detail with reference to the accompanying drawings. Unless otherwise specified, the features of the following embodiments and implementations can be combined with each other.

[0059] The present invention provides a tag calculation method based on hybrid expert LLM, comprising:

[0060] (1) Obtain user information and build a corresponding tag library;

[0061] User information definition: The user mentioned in this invention refers to the subject object (including individuals, items, cities, enterprises, groups, etc.) to be calculated by the tag. User information is the text description data related to the subject object.

[0062] Specifically, user information includes: resume information for individuals, city profile information for cities, news and public opinion documents for enterprises, and product review data for items, etc. After processing this user information using algorithms, specific tags can be associated with each user. For example, by using resume text information and a professional skills tag library, professional skills tags matching a resume can be obtained; by using city profile text data and a medicinal herb planting tag library, medicinal herb planting tags matching that city can be obtained; by using enterprise news and public opinion document information and an enterprise public opinion tag library, public opinion tags matching that enterprise can be obtained; and by using product review data and a sentiment evaluation tag library, sentiment evaluation tags matching that product can be obtained. Those skilled in the art should understand that other types of user information can also be obtained and corresponding tags constructed to obtain tags matching users; these will not be elaborated upon here.

[0063] (2) Based on user information and tags in the tag library, prompt1 is formed to ask questions to M different LLMs in the first hybrid expert model system. The returned result is constrained by a list of tags, and all results are merged to obtain the first tag set; the first hybrid expert model system includes M different LLMs;

[0064] In one embodiment, each tag in prompt1 is independently and randomly sampled from a tag library.

[0065] In one embodiment, the returned result is constrained to include a tag list in which the tags returned by each LLM are sorted from most important to least important, and the top N tags are returned.

[0066] In one embodiment, the method further includes expanding the tag library, specifically:

[0067] According to the tag list generated by LLM in the first hybrid expert model system, if the frequency of the generated tag is greater than the fourth preset threshold and / or the similarity between the generated tag and the tags in the tag library is greater than the fifth threshold, then the tag is added to the tag library.

[0068] (3) Calculate the similarity between each tag in the first tag set and the tags in the tag library. Each tag in the first tag set returns a set of similar tags, thus obtaining the second tag set;

[0069] In one embodiment, returning a set of similar tags includes: selecting at most K tags based on similarity and a third preset threshold.

[0070] (4) Convert the tags in the second tag set into vectors for the retrieval query question, and retrieve the vector database to obtain the context information most relevant to each tag; the vector database stores user information;

[0071] In one embodiment, the method further includes expanding the vector database, specifically:

[0072] The semantic information is extended based on the acquired user information, and the extended semantic information is vectorized and stored in the vector database; or a new vector database is established, and the extended semantic information is vectorized and stored in the newly established vector database.

[0073] (5) Insert the context information and its corresponding label into prompt2, and ask N different LLMs in the second hybrid expert model system to determine whether the label is correct or not; if the LLM determines that it is correct, then assign a weight of 1 / N; if it determines that it is incorrect, then assign a weight of 0; if the sum of the weights is greater than or equal to the first preset threshold, then retain the label; if it is less than or equal to the second preset threshold, then discard the label; if it is between the first preset threshold and the second preset threshold, then mark the label as a label to be determined.

[0074] If there is a label to be determined, input the label to be determined into the classification model and determine whether the label is correct or not. If it is correct, keep the label; otherwise, discard the label.

[0075] The second hybrid expert model system contains N different LLMs.

[0076] In one embodiment, the classification model is trained as follows:

[0077] Using at least one LLM in the second hybrid expert model system as the teacher model and the classification model as the student model, the classification model is trained by distillation.

[0078] In one embodiment, the system further includes a reputation score for the LLM in the second hybrid expert model system;

[0079] The reputation score is calculated as follows: R-LLM(i) = sum(LLM(i) == nLLM Output) / cumulative number of decisions; where LLM(i) == nLLM Output indicates whether the single output of the i-th LLM is equal to the final output. If it is, the output is 1; otherwise, it is 0. If the reputation score is lower than the third preset threshold, the LLM is removed from the label calculation. In some embodiments, the second hybrid expert model system has multiple LLMs (e.g., 8). If only one or two LLMs are removed from the label calculation, it does not affect the label calculation and no other processing is required. Of course, new LLMs can also be added to the second hybrid expert model system to replace the LLMs that are removed from the label calculation.

[0080] This invention innovatively combines LLM with a locally self-trained fine-tuning model, generative and discriminative label calculation, and prior and posterior computation in its label calculation method. Regarding the computational approach, a hybrid expert system model architecture is designed, employing parallel computation and model integration of multiple LLMs. Specifically, methods such as vector similarity-based semantic label expansion, prior computation based on LLM and few-sample prior knowledge, posterior computation based on LLM and RAG, label filtering based on multi-LLM joint soft decision, and secondary label recall based on knowledge distillation improve the accuracy of label calculation.

[0081] The present invention also provides a hybrid expert LLM and privacy-preserving trusted smart tag computing device, including a user side and a data computing sandbox;

[0082] The user side includes a local database and a vector database, both of which store encrypted user information;

[0083] The data computation sandbox is used to execute the aforementioned tag computation method based on hybrid expert LLM, obtain the retained tags, encrypt the tags, and output them to the user side for storage; the user information obtained is obtained by retrieving the specified user encrypted information from the local database and decrypting it.

[0084] Specifically, the user side establishes a two-way TLS authentication connection channel with the data computing sandbox. The user side transmits the user information that needs to be calculated in encrypted form to the data computing sandbox, and decrypts it in the memory of the sandbox environment to obtain the user's plaintext data.

[0085] In one embodiment, a regulator is also included to conduct an independent and public audit of the computational data, processes, and results.

[0086] Specifically, regulatory nodes typically select trusted third parties independent of the data owner and the computing party. On the one hand, this serves as a third-party oversight mechanism, enabling independent and public audits of computing data, processes, and results, thereby enhancing mutual trust and collaboration among consortium blockchain members. On the other hand, from a technical perspective, building a three-party system offers a higher degree of decentralization for consortium blockchains compared to a two-party system, further enhancing the security of the blockchain system itself.

[0087] This invention also designs a support architecture based on blockchain technology and a data computing sandbox to conduct comprehensive, trustless, and continuous credibility verification of the data, process, results, and model reputation of tag computing, thereby improving the effectiveness and credibility of tag computing and achieving data privacy protection.

[0088] The overall calculation method is as follows Figure 1The participating parties include data holders, computers, and regulators. The node layout consists of user-side nodes (corresponding to user data holders), data computation sandbox nodes (corresponding to computers), and regulator nodes, forming a blockchain network based on a consortium blockchain.

[0089] The Data Computation Sandbox creates a Secure and Trusted Execution Environment (TEE) to ensure the confidentiality, privacy, and integrity of tag computation and data processing. All plaintext information is decrypted and computed only within the sandbox's memory. The Data Computation Sandbox incorporates a hybrid expert model composed of multiple LLMs, integrating and combining the outputs of these multiple LLM models.

[0090] The blockchain network supports the establishment of a decentralized trust mechanism between the data computing sandbox and user-side data nodes. Four core smart contracts are deployed and executed in the blockchain: computing logic contract, computing process contract, credibility calculation, and LLM model reputation scoring contract, to ensure the credibility of the sandbox computing logic, process, results, and model.

[0091] The computation logic contract serializes the tag calculation algorithm code and data template (such as a prompt template) into binary code B, calculates the hash value of B to obtain Hash(B), and writes it into the blockchain's built-in ledger database during the blockchain initialization phase, providing query and verification interfaces. Before each execution of the tag calculation program, the data computation sandbox serializes the code to be executed, B', calculates its hash, and writes Hash(B') into the blockchain.

[0092] The computation process contract: On the user side, the hash value (denoted as Hash(R1)) of the plaintext original information (such as resume data) is written to the blockchain. The hash value (R2) of the decrypted plaintext original information (resume data) in the data computation sandbox memory is also written to the blockchain. Simultaneously, the hash value of the input information for each LLM is stored in an array list L = list[hash(LLM-n-input-i)], where LLM-n-input-i represents the i-th input content of the n-th LLM. The hash array L is then further written to the blockchain.

[0093] Trustworthiness calculation contract: Based on blockchain ledger data, executes the following verification logic:

[0094] Verification 1: Hash(B) == Hash(B')

[0095] Verification 2: Hash(R1) == Hash(R2);

[0096] Verification 3: All element values ​​in the array list L = list[hash(LLM-n-input-i)] are equal.

[0097] If the above three verification rules pass the smart contract execution verification, it proves that the entire label calculation process is credible; otherwise, the calculation result may be unreliable.

[0098] LLM Model Reputation Scoring Contract: Based on the aforementioned label calculation method, the "hybrid expert soft-judgment label filtering" calculation stage integrates the calculation results of multiple LLMs and runs a smart contract to score the reputation of each individual LLM. The reputation calculation method for each LLM is: R-LLM(i) = sum(LLM(i) == nLLM Output) / cumulative number of judgments. Here, LLM(i) == nLLM Output indicates whether the single output of the i-th LLM equals the final output (the output after multiple LLM voting soft-judgment and the classification model output). If it does, the output is 1; otherwise, it is 0. Then, the sum is accumulated and divided by the total number of judgments to obtain the reputation score of each LLM. For LLMs with reputation scores below a certain threshold, a certain proportion can be dynamically selected for removal, triggered and controlled by the smart contract. Figure 1 The multi-channel LLM fuse switch in the system enables the fuse to trip low-credit LLMs, thus preventing them from participating in tag calculation.

[0099] Example 1:

[0100] (1) System computing environment preparation.

[0101] like Figure 1 As shown, the main computing environment and technical preparations involved in this invention include:

[0102] Sandbox Environment: The data sandbox is a Trusted Execution Environment (TEE) built on an environment of isolation, providing a computing environment with CPU and memory isolation. Optional Trusted Execution Environment technologies include Intel SGX and ARM TrustZone. For example, a TEE based on Intel SGX / TDX1 can have a secure enclave capacity of up to 512GB per CPU, or a total capacity of 1TB for dual-socket systems, which can meet the execution space requirements of current large-scale models with hundreds of billions of processors.

[0103] LLM models: Large models such as OpenAI, Llama2, Llama3, and Qwen2 can be selected without limitation; considering data privacy protection, large models are deployed locally. The integration framework for large model applications can be open-source frameworks such as LangChain.

[0104] Embedding Model: You can choose Word2Vec, FastText, BERT, M3E: MokaMassive Mixed Embedding, OpenAI text-embedding-3-small, OpenAI text-embedding-ada-002, etc. The specific implementation depends on the parameter scale, resource requirements and accuracy.

[0105] Vector databases: Milvus, Annoy / Chroma / Faiss (lightweight), GaussDB Vector / Tencent Cloud VectorDB, etc. can be selected based on the data size and performance requirements.

[0106] Consortium blockchains: Commonly used open-source frameworks such as IBM Hyperledger Fabric can be selected.

[0107] (2) Data preparation

[0108] The data sources mainly include: resume text data (tens of thousands of resume documents) and an initial tag library (mainly including skill data, with tens of thousands of job skill tags already available). Examples of some tags are shown in Table 1.

[0109] Table 1: Examples of Some Job Skill Labels

[0110] 0 Python 1 MySQL 2 Django 3 Redis 4 Java 5 rear end 6 Golang 7 Computer-related majors 8 PyTorch 9 Docker 10 Oracle 11 reptile 12 Flask 13 Machine Learning / Algorithm Engineering 14 C / C++ 15 TensorFlow 16 Linux development / deployment experience 17 Pandas 18 MongoDB 19 Distributed experience 20 Microservice experience

[0111] Resume data is sensitive and is encrypted before being stored in the user's local database. Simultaneously, the resume data is also converted and synchronously stored in a vector database (KV database), where K represents vector information calculated from plaintext; V represents the text fragment corresponding to the vector, which is converted to ciphertext and stored in the vector database.

[0112] Since the tag library data does not involve sensitive user information, it is directly loaded and stored in the vector database.

[0113] The resume is long text data, which needs to be segmented or divided into sentences according to different user IDs (personnel), and then converted into vectors and stored in the vector database; the tags in the tag library are short texts, and each tag is converted into a vector and directly stored in the vector database.

[0114] (3) Privacy data transmission.

[0115] The encrypted data (resume information) of the specified user is passed to the data computing sandbox. The encrypted information of the user is decrypted in the memory of the data computing sandbox, and the plaintext resume data information of the user is obtained in the memory.

[0116] (4) Initial recall of labels

[0117] Based on the user's plaintext resume data and independently randomly sampled tag examples, a prompt language template prompt1 is formed. This prompt is used to ask multiple different LLMs (only two are used as examples in this embodiment, and no limit is imposed) at the same time. The returned result is constrained by a tag list (each LLM is sorted from the most important to the least important and the top N tags are returned; the example takes 4 tags).

[0118] Example of a prompt language template (prompt1) Figure 2 and Figure 3 As shown (for privacy reasons, the resume data here has been simplified; the actual resume text is longer).

[0119] like Figure 2 , Figure 3 As shown, hints are provided from two different LLMs (llama3 and qwen2), and the tag sets output by the two LLMs are not exactly the same.

[0120] Figure 2 The job skill tags "Python, Data Analysis, Software Development, Project Management, Human Resource Management, and Sales" in the prompt message Figure 3 The prompts “large model, autonomous driving, satellite navigation, FPGA, marketing, and financial management” are derived from two completely independent random samples of the initial tag library (sampling 6 tags as examples for each sample).

[0121] Finally, the result sets of labels generated by the two-way LLM are merged (union) to obtain a set S containing the following 5 labels: [Algorithm Development, CNN, RNN, JAVA, GO].

[0122] (5) Tag normalization and expansion

[0123] Based on the similarity between embedded vectors, the set S is compared with the vectors in the initial tag library using similarity calculation (cosine similarity is optional), and the top K tags (K is set to 8 for example) are retained. The expanded set is illustrated below:

[0124] Set K: [Algorithm Engineer, Deep Learning, Machine Vision, Java, Go, Software Development, Python, C]

[0125] After tag normalization and expansion, it can be seen that some skill terms are more standardized, such as "java" being normalized to "java language". However, while expanding the tags, some skill tags that may not be available or may be incorrect were recalled (such as C language, although C language and JAVA language have some similarities). Further testing of the expanded tags is needed in the next step.

[0126] (6) Tag verification based on RAG retrieval

[0127] The skill tags to be tested, such as "algorithm engineer" and "PYTHON language", are converted into vectors for retrieval query questions. The vector library of the corresponding user's original resume information is retrieved. The most relevant document fragments are selected by vector similarity calculation. Then, the matching document fragments are inserted into the template of the Improvement Project Prompt2 (the template is obtained by executing a blockchain smart contract). New query instructions are dynamically generated and asked simultaneously to LLM1 / LLM2.

[0128] A prompt language template (prompt2) template is as follows: Figure 4 As shown, P1 is the enhanced contextual information retrieved after matching from the vector database, and P2 is the label to be tested. After insertion, specific query instructions are dynamically generated, such as... Figures 5-8 As shown. Figure 5 As shown, the factual information of the dynamic query command, "has many years of experience in algorithm development projects," is the enhanced contextual information used for retrieval after matching from the vector database.

[0129] (7) Label filtering based on hybrid expert soft decision

[0130] The set K in step (6) has 8 labels to be tested: [Algorithm Engineer, Deep Learning, Machine Vision, JAVA, GO, Software Development, PYTHON, C]

[0131] As mentioned earlier, the tag filtering logic is divided into the following three cases: 1. If both check1 and check2 are negative, output "No". 2. If both check1 and check2 are positive, output "Yes". 3. Other cases (i.e., one outputs "Yes" and the other outputs "No"; in this embodiment, N=2, so the weight of other cases is 0.5. In this embodiment, the first preset threshold is 0.7 and the second preset threshold is 0.3. If the tag is between the first and second preset thresholds, it is marked as a tag to be determined), output "To be determined". After this step, tags with an output of "Yes" can be directly output to the next stage as confirmed tags. Tags with an output of "No" are directly discarded (indicating that the tag's credibility is insufficient).

[0132] The set after filtering set K is divided into 3 sets as follows:

[0133] K1: [Algorithm Engineer, Deep Learning, Machine Vision, Java, Go]

[0134] K2: [Software Development]

[0135] K3: [Python and C programming languages]

[0136] All labels in set K1 passed the test, labels in set K3 failed the test and were discarded, and labels in set K2 require further testing.

[0137] (8) Tag-based secondary recall based on knowledge distillation

[0138] First, a local classifier model is constructed using a teacher-student training paradigm. LLM1 / LLM2 acts as the teacher model, and the local classifier model serves as the student model. The local classifier model is modeled as a text matching problem between the question and the evidence. Fine-tuning is then performed on a pre-trained BERT model (other pre-trained language models can also be used, without limitation here). The core method is illustrated below. Figure 9 As shown.

[0139] Once the local student model is trained, the labels in the K2 set from the previous step can be further examined. If the output result is "yes", a secondary recall can be performed; if it is "no", they can be discarded.

[0140] After this step, the final result label set R (containing 6 valid labels) is obtained as follows:

[0141] R: [Algorithm Engineer, Deep Learning, Machine Vision, Java, Go, Software Development]

[0142] (9) Output of Results

[0143] The final output label results are encrypted and then sent to the user side for storage via a data security gateway.

[0144] Example 2: Tagging the city's natural resources for use in the field of smart cities.

[0145] Based on the descriptive information in the city's archival texts, tags for the medicinal plant resources possessed by the city are extracted. This can be used for the analysis and decision-making regarding medicinal plant production areas and urban resources.

[0146] The implementation steps are as follows (9 steps):

[0147] (1) System computing environment preparation.

[0148] Same as described in Example 1.

[0149] (2) Data preparation

[0150] The data sources mainly include: text data from city archives (such as archival information from hundreds of cities in China), and an initial Chinese medicinal herb resource tag library (mainly including the names of Chinese medicinal plants, with thousands of tags already existing). Some examples of tags are shown in Table 2.

[0151] Table 2: Sample Labels for Some Chinese Medicinal Herbs Cultivation

[0152] 0 Wuzhi rock ginger 1 Five-finger orange 2 Dongbai tea 3 mat grass 4 Juyan Tea 5 Plantain 6 Southern Gentian 7 Chicken bone grass 8 Golden Buddha Grass 9 Golden Grass 10 Golden Bull Grass 11 Dandelion 12 flat wood 13 bitter gentian 14 Cistanche deserticola 15 He Shou Wu 16 Huai Niu Xi 17 Black aconite 18 Polygonatum 19 Astragalus 20 Coptis chinensis 21 Coptis chinensis

[0153] The city archive data of the user (government department) is sensitive data and can be encrypted and stored in the user's local database. At the same time, the city archive text data will also be converted and synchronously stored in a vector database (KV database), where K is the vector information calculated based on plaintext; V is the text fragment corresponding to the vector, which is converted into ciphertext and stored in the vector database.

[0154] Since the tag library data does not involve sensitive user information, it is directly loaded and stored in the vector database.

[0155] The city archive text is long text data, which needs to be segmented or divided into sentences according to different city IDs, and then converted into vectors and stored in the vector database; the tags in the tag library are short texts, and each tag is converted into a vector and directly stored in the vector database.

[0156] (3) Privacy data transmission.

[0157] The encrypted data (city archive text) of the specified user is passed to the data computing sandbox. The encrypted information of the user is decrypted in the memory of the data computing sandbox, and the plaintext city archive text data of the user is obtained in the memory.

[0158] (4) Initial recall of labels

[0159] By combining the user's plaintext city archive text data and independently randomly sampled label examples, a prompt language template prompt1 is formed. This prompt is then used to ask multiple different LLMs (only two are used as examples here) simultaneously. The returned result is constrained to a list of labels (each LLM returns the top N labels sorted by importance from largest to smallest).

[0160] Figure 10 and Figure 11 This is an example of a prompt language template (prompt1) provided in Embodiment 2 of the present invention.

[0161] like Figure 10 and Figure 11 As shown, hints are provided from two different LLMs (llama3 and qwen2), and the tag sets output by the two LLMs are not exactly the same.

[0162] Finally, the result sets of labels generated by the two-way LLM are merged (union) to obtain a set S containing the following 10 labels: [Atractylodes macrocephala, Paeonia lactiflora, Corydalis yanhusuo, Scrophularia ningpoensis, Chamaejasminoides var. dahurica, Citrus reticulata, Citrus medica, Chamaejasminoides var. dahurica, Zingiber officinale, Hemerocallis fulva].

[0163] (5) Tag normalization and expansion

[0164] Based on the similarity between embedded vectors, the set S is compared with the vectors in the initial tag library using a similarity calculation (optionally cosine similarity), and the top K tags are retained. The expanded set is illustrated below:

[0165] Set K: [Atractylodes macrocephala, Paeonia lactiflora, Corydalis yanhusuo, Scrophularia ningpoensis, Dongbai tea, Asparagus cochinchinensis, Juyan tea, Daylily, Wuzhiyan ginger, Wuzhi citrus]

[0166] Among them, "Five-Finger Ginger" was normalized to the standard name "Five-Finger Rock Ginger" through semantic similarity calculation. "Buddha's Hand" was not in the initial Chinese medicinal material resource tag library, but after semantic expansion based on the corpus, it was mapped to the existing tag "Five-Finger Citrus" in the tag library.

[0167] Note: Since medicinal herbs often have alternative names, and the semantic correlation between these alternative names and their original names is not high, a specialized medicinal herb corpus / dictionary or knowledge graph can be used to assist in semantic expansion of medicinal herb names. The medicinal herb corpus information (e.g., the alternative names for Buddha's Hand are Buddha's Hand Citron and Five-Finger Citrus) should also be vectorized and stored in a vector database using the aforementioned method to support the contextual information for subsequent prompting processes.

[0168] (6) Tag verification based on RAG retrieval

[0169] The skill tags to be tested, such as "Wuzhiyan ginger" and "Wuzhi tangerine", are converted into vectors for retrieval query questions. The vector library of corresponding original city archive information is then retrieved. The document fragments most relevant to the query are selected through vector similarity calculation. The matching document fragments are then inserted into the template of the Improvement Project Prompt2 (the template is obtained by executing a blockchain smart contract). New query instructions are dynamically generated and asked simultaneously to LLM1 / LLM2.

[0170] A prompt language template (prompt2) template is as follows: Figure 12 As shown, P1 is the contextual information used for enhancement after matching from the city archive vector database, P1a is the contextual information used for enhancement from another source after matching from the medicinal herb corpus vector database, and P2 is the label to be tested. After insertion, specific query instructions are dynamically generated. For example... Figures 13-16As shown, some factual information in the dynamic query command is used to retrieve enhanced contextual information after matching with the city archive vector database, while other factual information, such as "Buddha's Hand is also known as Buddha's Hand Citron and Five-Finger Citrus," is used to retrieve enhanced contextual information after matching with the vector database of medicinal materials corpus.

[0171] like Figures 13-16 As shown, taking the two labels "Wuzhiyan ginger" and "Wuzhi tangerine" as examples, multiple LLMs produce consistent outputs for the label verification of "Wuzhi tangerine", but inconsistent outputs for the label verification of "Wuzhiyan ginger".

[0172] (7) Label filtering based on hybrid expert soft decision

[0173] The set K in step (6) has 10 labels to be tested: [Atractylodes macrocephala, Paeonia lactiflora, Corydalis yanhusuo, Scrophularia ningpoensis, Dongbai tea, Sedge grass, Juyan tea, Daylily, Wuzhiyan ginger, Wuzhi citrus]

[0174] As mentioned earlier, the tag filtering logic has three cases: 1. If both check1 and check2 are no, output "No". 2. If both check1 and check2 are yes, output "Yes". 3. Other cases (same as in Example 1), output "Pending confirmation". After this step, tags with an output of "Yes" can be directly output to the next stage as confirmed tags. Tags with an output of "No" are discarded (indicating insufficient tag credibility).

[0175] The set after filtering set K is divided into 3 sets as follows:

[0176] K1: [Atractylodes macrocephala, Paeonia lactiflora, Corydalis yanhusuo, Scrophularia ningpoensis, Dongbai tea, Asparagus cochinchinensis, Juyan tea, Daylily, Citrus aurantium]

[0177] K2:[Wuzhiyan Ginger]

[0178] K3:[]

[0179] All labels in set K1 passed the test, labels in set K3 failed the test and were discarded, and labels in set K2 require further testing.

[0180] (8) Tag-based secondary recall based on knowledge distillation

[0181] First, a local classifier model is constructed using a "Teacher-Student Training Paradigm" architecture. LLM1 / LLM2 acts as the teacher model, and the local classifier model acts as the student model. The local classifier model is modeled as a text matching problem between the question and the evidence, and fine-tuned based on the BERT pre-trained model (other pre-trained language models can also be used, which is not limited here).

[0182] Once the local student model is trained, the labels in the K2 set from the previous step can be further examined. If the output result is "yes", a secondary recall can be performed; if it is "no", they can be discarded.

[0183] After this step, the final result label set R (containing 10 valid labels) is:

[0184] R: [Atractylodes macrocephala, Paeonia lactiflora, Corydalis yanhusuo, Scrophularia ningpoensis, Dongbai tea, Asparagus cochinchinensis, Juyan tea, Daylily, Wuzhiyan ginger, Wuzhi citrus]

[0185] (9) Output of Results

[0186] The final output label results are encrypted and then sent to the user side for storage via a data security gateway.

[0187] The above embodiments are only used to illustrate the design concept and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, all equivalent changes or modifications made based on the principles and design ideas disclosed in the present invention are within the protection scope of the present invention.

Claims

1. A tag calculation method based on hybrid expert LLM, characterized in that, include: Obtain user information and build a corresponding tag library; Based on user information and tags in the tag library, prompt1 is formed, which is used to ask questions to M different LLMs in the first hybrid expert model system. The returned results are constrained by a list of tags, and all results are merged to obtain the first tag set. The similarity between each tag in the first tag set and the tags in the tag library is calculated, and each tag in the first tag set returns a set of similar tags, thus obtaining the second tag set; The tags in the second tag set are converted into vectors for the retrieval query question, and the vector database is searched to obtain the most relevant context information for each tag; the vector database stores user information. Insert the context information and its corresponding label into prompt2, and ask N different LLMs in the second hybrid expert model system to determine whether the label is correct or not; if the LLM determines that it is correct, it assigns a weight of 1 / N; if it determines that it is incorrect, it assigns a weight of 0; if the sum of the weights is greater than or equal to the first preset threshold, the label is retained; if it is less than or equal to the second preset threshold, the label is discarded. If the value falls between the first preset threshold and the second preset threshold, the label is marked as a label to be determined. If there is a label to be determined, input the label to be determined into the classification model and determine whether the label is correct or not. If it is correct, keep the label; otherwise, discard the label.

2. The method according to claim 1, characterized in that, Each tag in prompt1 is obtained by independent random sampling from the tag library.

3. The method according to claim 1, characterized in that, The returned results are constrained to include a list of tags: the tags returned by each LLM are sorted from most important to least important, and the top N tags are returned. The returned set of similar tags includes: sorted by similarity from largest to smallest and taking at most the top K tags based on a third preset threshold.

4. The method according to claim 1, characterized in that, The user in the user information refers to the subject object of the tag to be calculated, and the user information is the text description data related to the subject object.

5. The method according to claim 1, characterized in that, The classification model was trained as follows: Using at least one LLM in the second hybrid expert model system as the teacher model and the classification model as the student model, the classification model is trained by distillation.

6. The method according to claim 1, characterized in that, This also includes expanding the tag library, specifically: According to the tag list generated by LLM in the first hybrid expert model system, if the frequency of the generated tag is greater than the fourth preset threshold and / or the similarity between the generated tag and the tags in the tag library is greater than the fifth threshold, then the tag is added to the tag library.

7. The method according to claim 1, characterized in that, This also includes extending the vector database, specifically: The semantic information is extended based on the acquired user information. The extended semantic information is then vectorized and stored in the vector database, or a new vector database is created and stored in the newly created vector database.

8. The method according to claim 1, characterized in that, It also includes credit scoring of LLMs in the second hybrid expert model system; The reputation score is calculated as follows: R-LLM(i) = sum (LLM(i) == nLLM Output) / cumulative number of decisions; where LLM(i) == nLLM Output means determining whether the single output of the i-th LLM is equal to the final output. If it is, the output is 1, otherwise it is 0. If the reputation score is lower than the sixth preset threshold, the LLM will be removed from the label calculation.

9. A hybrid expert LLM and privacy-preserving trusted smart tag computing device, characterized in that, Including the user side and the data computing sandbox; The user side includes a local database and a vector database, both of which store encrypted user information; The data computing sandbox is used to execute the method described in claim 1, obtain a retained tag, encrypt the tag, and output it to the user side for storage; wherein the obtained user information is obtained by retrieving specified user encrypted information from the local database and decrypting it.

10. The apparatus according to claim 9, characterized in that, It also includes regulators, who are responsible for conducting independent and public audits of the calculation data, processes, and results.

Images