Security force conversion method and system based on large model

Abstract

The invention belongs to the technical field of local resource strength guarantee. A guarantee power conversion method based on a large model comprises the following steps: acquiring local resource power guarantee data to obtain a professional field data set; performing pre-training on a ChatGLM-6B model by using the professional field data set to obtain a pre-trained model; performing supervision adjustment on the pre-training model according to preset prompt and answer samples to obtain a supervised training model; performing RLHF parameter fine tuning on the supervised training model to obtain an RLHF training model; carrying out edge deployment on the RLHF training model, and integrating the RLHF training model with LangChain to obtain an RLHF interaction model; and the AI-agent is utilized to be combined with the RLHF interaction model, and guarantee force conversion is carried out. By taking the guarantee demand as traction, the local resource power is assisted in agile enabling and accurate energy release in the current fighting scene, and the local guarantee capability is improved.

Description

Technical Field

[0001] The present application belongs to the field of local resource force support technology, and in particular, relates to a support force conversion method and system based on a large model. Background Art

[0002] Artificial intelligence big models refer to machine learning models with super-large-scale parameters and super-powerful computing resources, which can process massive data and complete various complex tasks. With the launch of ChatGPT by OpenAI in 2022, various big models at home and abroad have sprung up like mushrooms after rain.

[0003] In response to the need for local resource support in conflict scenarios, it is urgent to address the current problems in the organization and use of resources and forces, carry out innovative research on the perception and use of local resource forces, and use support needs as a guide to help local resource forces to be swiftly empowered and accurately released in current conflict scenarios, thereby improving local support capabilities. Summary of the invention

[0004] Based on this, it is necessary to provide a large-model-based support force conversion method and system to address the above-mentioned technical issues.

[0005] In a first aspect, the present application provides a method for converting support forces based on a large model, the method comprising:

[0006] Obtain local resource support data and obtain professional field data sets;

[0007] Using the professional field dataset, pre-training the ChatGLM-6B model to obtain a pre-trained model;

[0008] According to the preset prompts and answer samples, the pre-trained model is supervised and adjusted to obtain a supervised training model;

[0009] Fine-tuning the RLHF parameters of the supervised training model to obtain the RLHF training model;

[0010] Deploy the RLHF training model on the edge and integrate it with LangChain to obtain the RLHF interaction model;

[0011] The AI-agent is used in combination with the RLHF interaction model to carry out the guarantee force conversion.

[0012] In some practicable methods, the step of using the professional field dataset to pre-train the ChatGLM-6B model to obtain the pre-trained model includes:

[0013] Performing structured division on the data in the professional field data set to obtain structured data, semi-structured data and unstructured data;

[0014] Converting the semi-structured data and combining it with the structured data to obtain a first initial corpus;

[0015] Using regular expressions and natural language processing tools, the unstructured data is processed to obtain a second initial corpus;

[0016] According to a preset division ratio, the target corpus is divided into two parts to obtain a training set and a validation set, wherein the target corpus includes the first initial corpus and the second initial corpus, and the first initial corpus and the second initial corpus are both texts;

[0017] Segmenting the data in the training set and the validation set respectively to obtain paragraphs or sentences corresponding to the text;

[0018] Using a word segmenter, segment the paragraph or sentence corresponding to the text to obtain words or subwords corresponding to the paragraph or sentence corresponding to the text;

[0019] Mapping the word or subword to an integer code to obtain an integer value corresponding to the word or subword;

[0020] Input the integer value corresponding to the training set into the initial word segmentation model to obtain an intermediate word segmentation model;

[0021] Inputting the integer value corresponding to the validation set into the intermediate word segmentation model, and adjusting the parameters of the intermediate word segmentation model according to the result output by the intermediate word segmentation model to obtain a target word segmentation model;

[0022] The vocabulary in the target word segmentation model is combined with the ChatGLM-6B original word segmentation model to obtain a new word segmentation model;

[0023] The ChatGLM-6B model is trained using the new word segmentation model to obtain a pre-trained model.

[0024] In some practicable methods, the step of performing supervised adjustment on the pre-trained model according to the preset prompts and answer samples to obtain the supervised training model includes:

[0025] Construct and obtain a number of preset samples with prompts and answers to obtain a preset training set;

[0026] In the preset training set, the prompt and answer of each preset sample are associated to obtain a continuous input sequence;

[0027] Using the markers, segmenting the prompt and the answer in the preset sample to obtain a prompt part and an answer part;

[0028] Using an autoregressive objective function, the loss of the prompt part corresponding to the user prompt is reset to zero to obtain the back propagation of the answer part;

[0029] According to the back propagation of the answer part, the parameters of the adjustable model part are adjusted, and the parameters of the frozen model part remain unchanged, so as to obtain a supervised training model, wherein the adjustable model part and the frozen model part constitute the pre-training model.

[0030] In some practicable methods, the step of performing RLHF fine-tuning on the supervised training model to obtain the RLHF training model includes:

[0031] Inputting a prompt into the supervised training model to obtain a plurality of candidate answers output by the supervised training model;

[0032] Obtaining ranking and scoring of the plurality of candidate answers output by the supervised training model to obtain a score for each candidate answer;

[0033] Extracting features of the candidate answers to obtain features of the candidate answers;

[0034] The input prompt and the candidate answer features are used as input, and the scores of the candidate answers are used as labels to form a reward sample, and a reward training set is obtained based on a plurality of the reward samples;

[0035] Using the reward training set, training an initial reward model to obtain a target reward model;

[0036] The target reward model is used to fine-tune the RLHF parameters of the supervised training model to obtain the RLHF training model.

[0037] In some practicable methods, the step of edge deploying the RLHF training model and integrating it with LangChain to obtain the RLHF interaction model includes:

[0038] Establishing an index for the professional domain dataset, wherein the index represents the division of text in the professional domain dataset into blocks to form document block vectors;

[0039] Obtaining an input prompt from a user, and converting the input prompt into a vector to obtain an input vector;

[0040] Compare the input vector with the document block vectors in the professional domain data set to obtain K document blocks with the highest similarity, wherein the K document blocks with the highest similarity serve as enhanced context information for the input prompt;

[0041] The K document blocks with the highest similarity are merged with the input prompt to obtain the prompt input into the RLHF training model, thereby forming an edge deployment of the RLHF training model.

[0042] In some practicable methods, the step of edge deploying the RLHF training model and integrating it with LangChain to obtain the RLHF interaction model includes:

[0043] Importing the LangChain library into the interactive environment formed by the user and the RLHF training model;

[0044] Initialize LangChain and create a LangChain instance connected to the RLHF training model to obtain the RLHF interaction model.

[0045] In some practicable ways, the step of using the AI-agent in combination with the RLHF interaction model to perform the step of ensuring force conversion includes:

[0046] Acquire a security requirement task, wherein the security requirement task includes security tasks of different categories;

[0047] According to the guarantee category, the guarantee demand task is decomposed by using AI-agent to obtain multiple guarantee category subtasks;

[0048] Inputting a plurality of the security category subtasks into the RLHF interaction model, executing them respectively, and obtaining output results of each of the security category subtasks;

[0049] According to the output results of the plurality of guarantee category subtasks, a guarantee force conversion is obtained to solve the actual demand guarantee prompts.

[0050] In a second aspect, the present application provides a large model-based support force conversion system, which is applied to the aforementioned large model-based support force conversion method, and the system includes:

[0051] The acquisition unit is used to obtain local resource force guarantee data and obtain professional field data sets;

[0052] A training unit, used to pre-train the ChatGLM-6B model using the professional field data set to obtain a pre-trained model;

[0053] A supervision unit is used to supervise and adjust the pre-trained model according to preset prompts and answer samples to obtain a supervised training model;

[0054] A fine-tuning unit, used for fine-tuning the RLHF parameters of the supervised training model to obtain the RLHF training model;

[0055] The interaction unit is used to deploy the RLHF training model on the edge and integrate it with LangChain to obtain the RLHF interaction model;

[0056] The result unit is used to utilize the AI-agent in combination with the RLHF interaction model to perform the guarantee force conversion.

[0057] In a third aspect, the present application provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the aforementioned large-model-based support force conversion method when executing the computer program.

[0058] In a fourth aspect, the present application provides a computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the aforementioned large-model-based support force conversion method.

[0059] Beneficial effects:

[0060] The present application provides a method for converting support forces based on a large model, which obtains local resource force support data and obtains a professional field data set; uses the professional field data set to pre-train the ChatGLM-6B model to obtain a pre-trained model; supervises and adjusts the pre-trained model according to preset prompts and answer samples to obtain a supervised training model; fine-tunes the RLHF parameters of the supervised training model to obtain an RLHF training model; deploys the RLHF training model at the edge and integrates it with LangChain to obtain an RLHF interaction model; uses AI-agent and combines it with the RLHF interaction model to convert support forces. ChatGLM-6B with a Chinese-English ratio of 1:1 is selected as the basic training model. It aims at the local resource force support needs in conflict scenarios, solves the problem of poor support capabilities of current resource forces in organizational use, and uses support needs as a guide to help local resource forces to be agilely empowered and accurately released in current dispute scenarios, thereby improving local support capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the conventional technology, the drawings required for use in the embodiments or the conventional technology descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0062] Figure 1 The present invention is a flowchart of a method for converting support forces based on a large model in one embodiment.

[0063] Figure 2The present invention is a flowchart of obtaining a pre-trained model for a method of converting support forces based on a large model in one embodiment.

[0064] Figure 3 The present invention is a flowchart of obtaining a supervised training model for a large-model-based support force conversion method in one embodiment.

[0065] Figure 4 The flowchart of the RLHF training model is obtained by a large-model-based support force conversion method in one embodiment.

[0066] Figure 5 The flowchart of the RLHF interaction model is obtained for a large-model-based support force conversion method in one embodiment. DETAILED DESCRIPTION

[0067] In order to facilitate understanding of the present application, the present application will be described more fully below with reference to the relevant drawings. Embodiments of the present application are provided in the drawings. However, the present application can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive.

[0068] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The term "and / or" used herein includes any and all couplings of one or more related listed items.

[0069] It can be understood that the terms "first", "second", etc. used in the present application can be used in this article to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from another element.

[0070] Some terms involved in this application are explained below to facilitate understanding of this application:

[0071] Token, usually refers to the basic unit of text data in the field of natural language processing (NLP) and machine learning.

[0072] Tokenizer is a basic tool in Natural Language Processing (NLP). Its main function is to split continuous text strings into smaller units, which can be words, phrases, subwords or symbols.

[0073] Subword is a concept used in natural language processing to represent smaller language units that make up a word.

[0074] The autoregressive objective function is a loss function used in generative models. Its core idea is to gradually predict each element in the sequence (such as words, characters, etc.) and complete the generation task of the entire sequence by step-by-step generation.

[0075] Backpropagation Algorithm is an optimization algorithm used to train artificial neural networks. It is used to calculate the gradient of each layer in the network and update the weights through the gradient descent method to minimize the loss function.

[0076] RLHF fine-tuning (Reinforcement Learning from Human Feedback fine-tuning) is a model optimization method that combines human feedback and reinforcement learning techniques. It is a model training technique that uses feedback from human evaluators to guide the model's learning process. During the learning process, reinforcement learning is a method that allows the agent to learn the best behavior strategy through interaction with the environment.

[0077] AI-agent refers to artificial intelligence agent, which is a software program implemented using artificial intelligence technology. It can perform automated tasks, make decisions, and interact with users or other systems.

[0078] SentencePiece is an open source word segmentation library developed by Google for text segmentation in natural language processing tasks. It supports word segmentation in multiple languages ​​and is particularly suitable for languages ​​without obvious word separation, such as Chinese, Japanese, and Korean.

[0079] like Figure 1 As shown, the first aspect of the present application provides a method for converting support forces based on a large model, the method comprising:

[0080] S100, obtain local resource support data and obtain professional field data sets.

[0081] Among them, local resource and force guarantee data refers to data related to the resource and force guarantee of a specific region or organization, which usually comes from data generated by government departments, social organizations, corporate records and historical archives. These data involve different professional fields, and the data in each field are aggregated to form data sets in different professional fields.

[0082] S200, using the professional field dataset, pre-training the ChatGLM-6B model to obtain a pre-trained model.

[0083] like Figure 2As shown, specifically, obtaining the pre-trained model includes the following steps:

[0084] S201, performing structured division on the data in the professional field data set to obtain structured data, semi-structured data and unstructured data.

[0085] Among them, the large model mentioned in this application is a language model, which is a model formed by reasoning about text. Since the text input into the language model is a string, and the string cannot be recognized by the language model, it is necessary to convert the text into numbers to express it. In this way, the data in the professional field data set is first structured and divided, and then converted into numbers. For professional field data, the professional field data involved is multi-heterogeneous, including structured and semi-structured data such as database tables, tables, and texts, and unstructured data such as pictures and voices. These different types of data form a corpus, i.e., a professional field data set. The corpus is mainly demand and guarantee data in different scenarios. Different scenarios have different demand categories, and different guarantee forces and methods. The basis for structured division of data in professional field data sets usually includes relatively fixed structures such as the organizational form and storage method of data. After defining the rules for structured division, those that meet the structured division rules are structured data, and those that partially meet the structured division rules are semi-structured data, and those that do not follow a fixed format are unstructured data, such as images and videos.

[0086] S202: Convert the semi-structured data and combine it with the structured data to obtain a first initial corpus.

[0087] After obtaining the semi-structured data in the aforementioned steps, in order to facilitate data processing in the subsequent steps, the semi-structured data needs to be converted into structured data. The conversion method can be to use regular expressions to extract specific patterns and data fragments in the semi-structured text, and then organize these fragments into a structured format, segment the text, identify the boundaries of the data records, and segment them into separate records, apply NLP technology, such as entity recognition and relationship extraction, to identify structured information in the text, and other conventional methods, which are not limited by this application. Next, the semi-structured data and the structured data are combined to obtain the first initial corpus.

[0088] S203: Process the unstructured data using regular expressions and natural language processing tools to obtain a second initial corpus.

[0089] Specifically, after obtaining the unstructured data in the aforementioned steps, conventional regular expressions and natural language processing tools are used to process the unstructured data. In addition, when the unstructured data includes images, videos or audio, conventional feature extraction methods are needed to extract features from the images, videos or audio. Next, regular expressions and natural language processing tools are applied to the extracted features to process the unstructured data into structured data, thereby obtaining a second initial corpus composed of structured data.

[0090] It should be noted that the first initial corpus and the second initial corpus are obtained in the above steps. The Chinese in the first initial corpus and the second initial corpus can be divided into common Chinese word segmentations and uncommon words. Common words are divided into one token, and uncommon words can be divided into 2-3 tokens as needed so that ChatGLM-6B can be trained.

[0091] S204, dividing the target corpus into two parts according to a preset division ratio to obtain a training set and a validation set, wherein the target corpus includes the first initial corpus and the second initial corpus, and the first initial corpus and the second initial corpus are both texts.

[0092] Specifically, according to a preset division ratio, such as a ratio of 8:2, the target corpus is divided into a training set and a validation set. Exemplarily, the training set: validation set = 8:2.

[0093] S205, segmenting the data in the training set and the validation set respectively to obtain paragraphs or sentences corresponding to the text.

[0094] Specifically, segmenting the data in the training set and the validation set may include the following steps:

[0095] Construct text segmentation rules;

[0096] According to the text segmentation rule, the data in the training set and the validation set are segmented to obtain the paragraphs or sentences corresponding to the text.

[0097] Among them, the text segmentation rules can be adjusted as needed. For example, the text segmentation rules may include: punctuation rules, paragraph separators, text structure, natural language processing tools, context information, and one or more methods to achieve paragraph or sentence segmentation of text. It should be noted that the above text segmentation method is only for illustrative purposes, and other segmentation methods, such as machine learning models, rule engines, etc., can also be used as needed.

[0098] Punctuation rules: use punctuation marks such as period (.), question mark (?), exclamation mark (!) as the basis for sentence segmentation;

[0099] Paragraph separators, which recognize separators between paragraphs, such as blank lines or paragraph marks, and divide the text into paragraphs;

[0100] Text structure, using structural elements in the text, such as headings, subheadings, lists, etc., to determine paragraph boundaries;

[0101] Natural language processing tools, such as spaCy, NLTK, or Stanford NLP, which include sentence boundary detection.

[0102] Contextual information,Considering contextual information, we can ensure that the segmentation of text does not destroy the coherence and readability of the text.

[0103] S206, using a word segmenter, segmenting the paragraphs or sentences corresponding to the text to obtain words or sub-words corresponding to the paragraphs or sentences corresponding to the text.

[0104] Specifically, in the aforementioned steps, when the paragraphs or sentences corresponding to the text are obtained, a conventional word segmenter is used to continue to segment the paragraphs or sentences corresponding to the text to form smaller units, such as words or subwords, so that in the subsequent steps, the ChatGLM-6B model can be trained using words or subwords.

[0105] S207, mapping the word or sub-word to an integer code to obtain an integer value corresponding to the word or sub-word.

[0106] Specifically, after obtaining the word or sub-word in the aforementioned steps, the process of mapping the word or sub-word to an integer code can be understood as vocabulary construction or vocabulary mapping, that is, first, construct a vocabulary, and assign a unique integer number to each entry in the vocabulary, and then map the word or sub-word to each entry so that each word or sub-word has a unique integer code, so that if described as a dictionary, the key is the entry and the value is the integer code, referred to as the integer value. It can be understood that after the above processing, each word or sub-word has a unique integer value, and when the word or sub-word is input into the corresponding model, it will be input in the form of an integer value so that the corresponding model can recognize it.

[0107] It should be noted that after a word or subword has a unique integer encoding, the word or subword in the validation set and the training set will be replaced by an integer value.

[0108] S208, inputting the integer value corresponding to the training set into the initial word segmentation model to obtain an intermediate word segmentation model.

[0109] Specifically, after the integer values ​​corresponding to the training set are obtained in the aforementioned steps, the integer values ​​of the training set are input into the initial word segmentation model, and the initial word segmentation model is trained. After the training is completed, the intermediate word segmentation model can be obtained.

[0110] It should be noted that the purpose of the intermediate word segmentation model is to build a model that can segment text into words or subwords. These segmented words or subwords can be further mapped to integer codes for pre-training of the word segmentation model. Specifically, the purpose of training the word segmentation model includes the following aspects:

[0111] The training of the word segmentation model helps generate a vocabulary (including words or subwords) that is customized for a specific corpus (training text). The vocabulary can better capture the language characteristics in the corpus, thereby improving the adaptability of the pre-trained model to the corpus.

[0112] Through the word segmentation model, segmenting the text into reasonable words or subwords helps to reduce the size of the vocabulary while retaining sufficient semantic information. This segmentation method can make the model more flexible when dealing with unknown words or variant words, thereby improving the generalization ability of the model.

[0113] The training of the word segmentation model ensures that word segmentation can be performed consistently when processing new text, thereby ensuring that the encoding method of the RLHF interaction model input is consistent during the pre-training and inference phases. This is very important for maintaining the prediction accuracy and stability of the RLHF interaction model.

[0114] S209: Input the integer value corresponding to the verification set into the intermediate word segmentation model, and adjust the parameters of the intermediate word segmentation model according to the result output by the intermediate word segmentation model to obtain a target word segmentation model.

[0115] Specifically, after obtaining the intermediate word segmentation model in the aforementioned steps, the intermediate word segmentation model needs to be verified. At this time, the integer value corresponding to the verification set is input into the intermediate word segmentation model to obtain the output result of the intermediate word segmentation model. The output result is used to judge the accuracy of the intermediate word segmentation model. When there are errors and other problems in the output result, the parameters of the intermediate word segmentation model are adjusted so that the target word segmentation model can be finally formed after the parameter adjustment.

[0116] S210, using the vocabulary in the target word segmentation model and merging it with the ChatGLM-6B original word segmentation model to obtain a new word segmentation model.

[0117] Specifically, after obtaining the vocabulary in the target word segmentation model, it is necessary to merge the vocabulary with the original ChatGLM-6B word segmentation model. It should be noted that the vocabulary in the target word segmentation model is merged with the vocabulary in the original ChatGLM-6B word segmentation model. If there are duplicate entries, delete the duplicate entries and keep one to avoid duplication. Next, re-encode the vocabulary in the new word segmentation model and assign an index to ensure that each entry in the new word segmentation model has a unique integer value.

[0118] S211, using the new word segmentation model, train the ChatGLM-6B model to obtain a pre-trained model.

[0119] Specifically, the purpose of training the ChatGLM-6B model using the new word segmentation model is to enable the ChatGLM-6B model to better adapt to the new vocabulary, thereby improving the performance of the ChatGLM-6B model in force support tasks. The purpose of this retraining includes the following aspects:

[0120] 1. Adapt to a new vocabulary

[0121] Vocabulary expansion: The merged vocabulary often contains new words or subwords that may not exist in the vocabulary of the original ChatGLM-6B model. Retraining the model allows it to learn how to use these new words effectively, so that it can perform better when dealing with new tasks or domains.

[0122] Encoding consistency: After retraining, the pre-trained model relearns the mapping between text and the new vocabulary, ensuring that the encoding of the text data matches the model’s input when using the new vocabulary.

[0123] 2. Improve the performance of pre-trained models in specific areas

[0124] Domain adaptability: If the new vocabulary comes from data in a specific field (for example, the professional field of support forces), retraining can help the model better grasp the language characteristics of that field, making its performance in related tasks more accurate and efficient.

[0125] Integration of new knowledge: Through pre-training, the pre-trained model can absorb new language patterns, terminology, and expressions, which helps to enhance the understanding and generation capabilities of the pre-trained model in specific application scenarios.

[0126] 3. Optimize the generation quality of pre-trained models

[0127] Naturalness of generated text: A new vocabulary may contain words or expressions that are more relevant to the target application. By retraining, the pre-trained model can better utilize these new words, thereby improving the naturalness and relevance of the generated text.

[0128] 5. Customize the pre-trained model

[0129] Customized pre-trained model: Through pre-training, ChatGLM-6B can be customized to make it more suitable for force support missions.

[0130] S300, according to preset prompts and answer samples, supervised adjustment is performed on the pre-trained model to obtain a supervised training model.

[0131] like Figure 3 As shown, specifically, the steps of obtaining the supervised training model include:

[0132] S301, construct and obtain a number of preset samples with prompts and answers to obtain a preset training set.

[0133] Specifically, a preset training set is constructed, which contains several samples with clear prompts and corresponding answers. This ensures that, in the case of uneven data diversity and quality, high-quality SFT data examples are used, including the quality of the output results. It should be noted that the answer corresponding to the prompt can be one or more.

[0134] S302, in the preset training set, associating the prompt and answer of each preset sample to obtain a continuous input sequence.

[0135] Specifically, after obtaining the preset training set in the aforementioned steps, it is necessary to associate the prompt and answer of each preset sample in the preset training set to form a continuous input sequence so that the pre-trained model can receive complete context information. When generating answers, the content of the prompt part can be directly referenced, thereby improving the accuracy and relevance of the generated answers. In addition, the input sequence is used as the initial input, so that the pre-trained model gradually generates the content of the answer part by autoregression. The prompt part of the input sequence provides the conditions for generating answers, so that the pre-trained model can make predictions based on the prompts at each step of generation. In addition, when the prompts and answers are clearly associated together, the pre-trained model can better understand which part is the prompt and which part is the answer that needs to be generated, reducing the possibility of ambiguity.

[0136] S303, using the markers, segmenting the prompt and the answer in the preset sample to obtain a prompt part and an answer part.

[0137] Specifically, in order to better segment the associated prompts and answers, a tag, such as a token (e.g., [SEP]), is used to separate the prompt and answer segments. In this way, using the tag, a sample is segmented into a prompt part and an answer part. Ensure that the pre-trained model can correctly understand the structure of the input, that is, the pre-trained model can clearly distinguish the part that needs to be predicted (i.e., the answer part) from the contextual information provided by the user (i.e., the prompt part).

[0138] S304, using an autoregressive objective function, the loss of the prompt part corresponding to the user prompt is reset to zero, and the back propagation of the answer part is obtained.

[0139] Specifically, the autoregressive objective function will gradually predict the corresponding answer part based on the given prompt part. In the specific implementation, the loss of the token from the user prompt is set to zero, indicating that these tokens will not affect the calculation of the loss function, and the pre-trained model only needs to focus on learning how to generate the correct answer part. In this way, in each training iteration, the pre-trained model will calculate the loss of the generated answer part and pass the loss back to the parameters of the adjustable model part through the back-propagation algorithm. In this process, the pre-trained model will gradually adjust its parameters to minimize the loss, thereby improving the accuracy and quality of answer generation.

[0140] S305, adjusting the parameters of the adjustable model part according to the back propagation of the answer part, and keeping the parameters of the frozen model part unchanged, to obtain a supervised training model, wherein the adjustable model part and the frozen model part constitute the pre-training model.

[0141] Specifically, the pre-trained model consists of two parts, one of which is the adjustable model part and the other is the frozen model part; the adjustable model part refers to the model parameter part that can be adjusted and optimized during the training process. Usually, it includes a part of the layers or certain parameters in the pre-trained model, which will be updated during the training process to minimize the loss function; the frozen model part refers to the pre-trained model parameter part that is not updated during the training process. These parameters remain unchanged during training and are not affected by the gradient. The parameters of the frozen model part help maintain the existing capabilities of the pre-trained model and avoid overfitting or destruction of existing knowledge in new task training.

[0142] S400, fine-tuning RLHF parameters of the supervised training model to obtain a RLHF training model.

[0143] like Figure 4 Specifically, the steps of obtaining the RLHF training model include:

[0144] S401, inputting a prompt into the supervised training model to obtain a plurality of candidate answers output by the supervised training model.

[0145] Specifically, after inputting a prompt into the supervised training model, the supervised training model will generate multiple candidate answers based on the input prompt, and multiple candidate answers may have different qualities and relevances. Therefore, the multiple candidate answers obtained need to be screened in subsequent steps.

[0146] S402, sorting and scoring the multiple candidate answers output by the supervised training model to obtain a score for each candidate answer.

[0147] Specifically, a scoring mechanism is constructed, and the scoring mechanism can score the candidate answers according to accuracy, fluency, relevance, etc. It should be noted that the scoring mechanism can be set according to actual conditions, and this application does not limit this. After obtaining the scoring mechanism, each candidate answer is scored according to the scoring mechanism, and the score represents the accuracy, fluency, relevance, etc. of the candidate answer.

[0148] S403: extract features of the candidate answers to obtain features of the candidate answers.

[0149] Specifically, feature extraction is performed on the candidate answers, and the feature extraction method used can be a conventional convolutional neural network or a conventional recurrent neural network. The present application does not limit the method of feature extraction. The features of the candidate answers may include one or more of linguistic features, semantic features, or statistical features. In addition, the extracted features are summarized as feature vectors (word embedding vectors), which represent the attributes and quality of the candidate answers.

[0150] It should be noted that step S403 can be adjusted to before S402, that is, first extract features from the candidate answers, and then score the multiple candidate answers.

[0151] S404, taking the input prompt and the features of the candidate answers as input, and the scores of the candidate answers as labels, to form a reward sample, and obtaining a reward training set based on a plurality of the reward samples.

[0152] Specifically, the input prompt is converted into a word embedding vector. For example, using methods such as Word2Vec and GloVe, the entire input prompt can be converted into a context-related vector representation using context-aware word vector generation methods (such as BERT and GPT). This method takes into account the different meanings of words in different contexts. When the candidate answer features are also word embedding vectors, the word embedding vector of the input prompt and the word embedding vector of the candidate answer are concatenated to form a comprehensive input vector. In this way, the comprehensive input vector has a corresponding score, and a reward sample is obtained. Next, multiple reward samples are aggregated to form a reward sample and a reward training set. This reward training set is used to train the reward model to learn the mapping relationship from prompt and answer features to score labels.

[0153] S405: Using the reward training set, train an initial reward model to obtain a target reward model.

[0154] Specifically, after obtaining the reward training set, the reward training set is used to train the initial reward model, and after the initial reward model is trained, it is converted into a target reward model. The purpose of the target reward model is to simulate human scoring of texts, and the target reward model predicts which score of human annotations is higher.

[0155] S406, using the target reward model, fine-tuning the RLHF parameters of the supervised training model to obtain the RLHF training model.

[0156] Exemplarily, a series of questions or prompts are input to the supervised training model so that the supervised training model generates multiple candidate answers. Next, the target reward model is used to score each candidate answer to evaluate the performance of these candidate answers in terms of accuracy, fluency, relevance, etc. According to the score of the target reward model, feedback information about the candidate answers is collected, which will serve as the basis for fine-tuning the supervised training model. Then, the score of the reward model is converted into a reward signal to guide the parameter update of the supervised training model. The supervised training model parameters are continuously updated in multiple iterations, and each iteration is adjusted according to the new feedback to improve the ability of the supervised training model to generate high-quality answers, and finally the RLHF training model is obtained.

[0157] S500, edge deploys the RLHF training model and integrates it with LangChain to obtain the RLHF interaction model.

[0158] like Figure 5 Specifically, the steps of obtaining the RLHF interaction model include:

[0159] S501, creating an index for the professional domain dataset, wherein the index represents dividing the text in the professional domain dataset into blocks to form document block vectors.

[0160] Specifically, indexing can greatly improve the efficiency of retrieving information in documents, especially when dealing with large amounts of data. Convert document chunks into vectors (using techniques such as word embedding and sentence embedding), and create vector indexes (for example, using libraries such as Annoy and Faiss) to speed up similarity searches.

[0161] It should be noted that indexing is usually performed offline, including data cleaning and extraction, converting different file formats (such as PDF\html\word, etc.) into plain text, then segmenting the text, and creating an index. The way to create the index is conventional, and this application does not limit how to create the index. Offline indexing can ensure the consistency and stability of the data. During the index construction process, the data will not change, so the accuracy and completeness of the index can be ensured. Offline indexing can optimize the use of computing resources, ensure data consistency, reduce system load, provide processing flexibility, and facilitate data preprocessing and system maintenance.

[0162] S502, obtaining an input prompt of the user, and converting the input prompt into a vector to obtain an input vector.

[0163] Specifically, after obtaining the user's input prompt, the user's input prompt (a piece of text) may be segmented into words or sub-words.

[0164] Convert each word or subword after segmentation into the corresponding word embedding vector.

[0165] The embedding vectors of all words or subwords are concatenated to form a long vector, the input vector; the long vector contains the information of the entire input prompt.

[0166] In the above manner, the input prompt can be converted into an input vector.

[0167] S503, comparing the similarity between the input vector and the document block vector in the professional field data set to obtain K document blocks with the highest similarity, wherein the K document blocks with the highest similarity serve as enhanced context information of the input prompt, K is a positive integer, and K can be adjusted according to specific application requirements.

[0168] Specifically, the input vector and the document block vector are compared for similarity using a vector similarity comparison method. The similarity comparison method is a conventional method, such as cosine similarity. After the comparison is completed, the K document blocks with the highest similarity are obtained. These K document blocks with the highest similarity are used as context information and merged with the user's input prompt to form a new prompt or context. The enhanced context information can provide more background information to help the RLHF training model generate more accurate and relevant answers.

[0169] S504: merge the K document blocks with the highest similarity with the input prompt to obtain a prompt input into the RLHF training model, thereby forming an edge deployment for the RLHF training model.

[0170] Specifically, after the user's input prompt is merged with the K document blocks, it can form richer prompt information, which can be more easily understood by the RLHF training model. Therefore, the K document blocks with the highest similarity are merged with the input prompt and used as the input prompt of the RLHF training model, so that the RLHF training model can provide more effective answers or suggestions in practical applications.

[0171] It should be noted that the edge deployment of the RLHF training model means that the RLHF training model is deployed at the edge of the network, that is, close to the data source or user, rather than a centralized cloud server. The distance that data is transmitted in the network is reduced, thereby reducing the response time and providing faster services. User privacy can be better protected because sensitive data does not need to be transmitted to a remote server. The reliability of the system is improved, and the edge device can operate independently even if the cloud service is unavailable.

[0172] In some examples, the step of obtaining the RLHF interaction model further includes:

[0173] S505, importing the LangChain library in the interactive environment formed by the user and the RLHF training model;

[0174] S506, initialize LangChain, and create a LangChain instance connected to the RLHF training model to obtain the RLHF interaction model.

[0175] Specifically, LangChain is an efficient framework designed to simplify and optimize the interaction with large models. The RLHF fine-tuning model obtained above is integrated into the application using LangChain's rich tools and interfaces. In this way, a chat is established using LangChain to conduct conversations in different scenarios. In addition, when LangChain and the RLHF fine-tuning model are combined, complex data analysis of local support forces can be performed and easy-to-understand reports or visualization results can be generated.

[0176] S600, utilizes AI-agent, combined with the RLHF interaction model, to perform security force conversion.

[0177] Specifically, S600 includes the following steps:

[0178] S601, obtaining a guarantee demand task, wherein the guarantee demand task includes different types of guarantee tasks.

[0179] S602, according to the guarantee category, using AI-agent to decompose the guarantee demand task to obtain a plurality of guarantee category subtasks.

[0180] S603, inputting a plurality of the security category subtasks into the RLHF interaction model, executing them respectively, and obtaining an output result of each of the security category subtasks.

[0181] S604, obtaining a support force conversion for solving the actual demand support prompt according to the output results of the plurality of support category subtasks.

[0182] Specifically, first, a support demand task containing multiple different categories was obtained through AI-agent. Then, based on the support categories of these tasks, AI-agent decomposed the complex support needs into multiple subtasks that are easier to manage and execute. Each subtask is then input into the RLHF interaction model for processing, and the RLHF interaction model generates corresponding output results for each subtask. Finally, combined with these output results, AI-agent conducts comprehensive analysis and conversion to form a set of solutions that can solve actual support needs, thereby achieving effective deployment of support forces and optimal allocation of resources. This process not only improves the efficiency and accuracy of task processing, but also enhances the ability to respond to complex support needs through intelligent decomposition and integration.

[0183] Example

[0184] A method for transforming support forces based on a large model includes the following steps:

[0185] 1. Collect local resource support data to form a professional field data set

[0186] The language model is to reason about the text. Since the text is a string, but for the large model, the input can only be numbers, so the text needs to be converted into numbers to express. First, it is necessary to collect domain knowledge to form a Chinese corpus in the professional field as a training data set. The corpus collected in this application is mainly demand and guarantee data in different scenarios. Different scenarios have different demand categories, and different guarantee forces and methods. The professional field data involved is multi-heterogeneous, including structured and semi-structured data such as database tables, tables, texts, and unstructured data such as pictures and voices to form a corpus.

[0187] 2. Use professional field data sets to pre-train the ChatGLM-6B model to obtain a pre-trained model

[0188] The corpus of the present invention includes both structured data and unstructured data.

[0189] (1) For the structured data collected in the corpus (training text), this application uses SentencePiece to train its own word segmentation model. First, it is necessary to divide each type of data into two parts, a training set and a validation set, of which 80% of the data is the training set and the remaining 20% ​​of the data is the validation set; secondly, classify, organize and clean each type of data in the training set, divide the data into paragraphs or sentences, use a word segmenter to divide the text into words or subwords, and then map these words to integer codes so that they can be used for pre-training. After the word segmentation model training is completed, the word segmentation model can be called using the validation set to test the effect of the word segmentation model;

[0190] (2) For unstructured data, regular expressions and natural language processing tools can be used for processing.

[0191] After testing, the trained word segmentation model is merged with the original word segmentation model of ChatGLM-6B to form a new word segmentation model in the field. The merged new word segmentation model is used to pre-train the ChatGLM-6B large model to obtain a pre-trained model.

[0192] 3. Perform supervised fine-tuning on the pre-trained model to obtain a supervised fine-tuning model

[0193] Due to the diversity and uneven quality of data, we first need to collect thousands of high-quality SFT data examples. During the fine-tuning process, each sample includes a prompt and an answer. The corresponding prompt in this application is the guarantee requirement, and the answer is the resource guarantee method corresponding to the guarantee requirement. First, all prompts and answers in the training set are concatenated, and a special token is used to separate the prompt and answer fragments. Using the autoregressive target, the token loss from the user prompt is zeroed, so only the answer token is back-propagated.

[0194] Model fine-tuning is divided into efficient parameter fine-tuning and full / partial parameter fine-tuning. This application adopts full / partial parameter fine-tuning, which means training directly on the model and fine-tuning some parameters of the model by freezing them. Exemplarily, the model can be fine-tuned twice to obtain a supervised fine-tuning model.

[0195] 4. Perform RLHF fine-tuning on the supervised fine-tuning model to obtain the RLHF fine-tuning model

[0196] (1) This step first requires training a reward model. In principle, the supervised fine-tuning model can be directly fine-tuned by domain experts based on the input and labeled output. However, this requires providing some samples to the domain experts and scoring them after each round of RLHF fine-tuning optimization. This process requires a lot of labor costs and a large number of data sets. Therefore, a simpler way is to train a reward model with the domain expert annotation set, and have the domain experts manually sort the different output results under the same input. The purpose of the reward model is to simulate human scoring of text, and the model predicts which human annotation has a higher score.

[0197] (2) Reinforcement learning based on human feedback: Based on the supervised fine-tuning model and reward model in the previous step, a reinforcement learning cycle is performed, which is mainly divided into the following three steps:

[0198] ① Domain experts label and generate feedback on the input.

[0199] ② Use a reward model to score feedback.

[0200] ③ Perform a round of reinforcement learning on the scoring strategy optimization.

[0201] After the above three steps, the RLHF fine-tuning model is obtained.

[0202] 5. Deploy the RLHF fine-tuned model

[0203] RAG technology is used to deploy large models at the edge. In the AI ​​large model, when answering questions or generating text, RAG first retrieves relevant information from a large document library, and then uses the retrieved information to generate responses or text, thereby improving the quality of predictions.

[0204] In this application, the RLHF fine-tuning model obtained in the previous step needs to be deployed, which mainly includes the following steps:

[0205] (1) Create an index: First, create an index. This process is usually performed offline and includes data cleaning and extraction, converting different file formats (such as PDF, HTML, Word, etc.) into plain text, then segmenting the text and creating an index.

[0206] (2) Retrieval: Use the same encoding model to convert the user input into a vector, calculate the similarity between the question embedding and the document block embedding, and select the top K document blocks with the highest similarity as the enhanced context information of the current question.

[0207] (3) The given question and related documents are combined into a new prompt, and the RLHF fine-tuned model then answers the question based on the provided information. If there is historical dialogue information, it can also be incorporated into the prompt for multi-round dialogue.

[0208] 6. Integrate the RLHF fine-tuning model with LangChain

[0209] LangChain is an efficient framework designed to simplify and optimize interactions with large models. This application uses LangChain's rich tools and interfaces to integrate the RLHF fine-tuning model obtained above into the application.

[0210] The connection between LangChain and the big model mainly includes the following steps:

[0211] (1) Import the LangChain library: Import the LangChain library in the interactive environment;

[0212] (2) Initialize LangChain: Create a LangChain instance to start interacting with the RLHF fine-tuned model obtained above;

[0213] (3) Configure the connection with the large model: configure it according to the RLHF fine-tuning model obtained above;

[0214] (4) Implement model interaction: Use LangChain instances to perform specific tasks, such as generating text, answering questions, etc.

[0215] In this application, taking the retrieval-based question answering task as an example, this task uses the RLHF fine-tuned model obtained above to complete automatic question answering for a specific document. The process includes: text reading, text segmentation, text / question vectorization, text-question matching, combining the matching text as context and question to generate the corresponding input as the input of the model, and generating the answer.

[0216] In this application, the functions that can be implemented using LangChain include:

[0217] (1) Use LangChain to quickly build a chatbot. By customizing the conversation style, it can achieve natural and smooth conversations in a variety of scenarios;

[0218] (2) Combining LangChain and the RLHF fine-tuning model, complex data analysis of local resource forces can be performed and easy-to-understand reports or visualization results can be generated.

[0219] 7. Apply AI-agent to RLHF fine-tuning model

[0220] Like the ChatGLM-6B model, the professional field RLHF fine-tuning model obtained in this application also has the analytical reasoning ability of a large model, can accept input, and can output text / media. However, it cannot have the ability to plan, think, and remember like humans. AI-agent can be understood as a general problem solver that has the ability to plan, think, remember, and use tool functions. It is a computer program that can independently complete a given task, provide accurate information retrieval services, maintain contextual coherence in complex conversations, and provide a smooth communication experience.

[0221] In this application, the professional field RLHF fine-tuning model obtained above is combined with AI-agent, and the RLHF fine-tuning model is equipped with AI-agent. The professional field RLHF fine-tuning model is regarded as the brain. AI-agent can think according to the task objectives, decompose the tasks, and then call the corresponding tools to complete the tasks. It has three key parts:

[0222] (1) Planning: In this application, generally, a security requirement involves many security categories. Taking emergency rescue as an example, the security categories may include road emergency repair security, rescue tool security, food security, medical and health resource security, etc. When receiving a security requirement, the AI-agent will first decompose the large security task into subtasks according to the security category, and plan the execution process of each security subtask; the AI-agent will think and reflect on the task execution process according to the task progress, and decide whether to continue to execute the task, or judge that the task is completed and terminate the operation.

[0223] (2) Memory: Short-term memory refers to the context in the process of executing a task. It will be generated and temporarily stored during the execution of the subtask and will be cleared after the task is completed; long-term memory is information retained for a long time, generally referring to an external knowledge base, which is usually stored and retrieved using a vector database. In this application, short-term memory mainly refers to the situation when two consecutive security demands are received, and the second security plan can be generated based on the decomposition and execution of the previous unfinished security task; long-term memory mainly refers to the security plan and execution status generated each time according to the security demand, which are saved to form a professional field knowledge base, so as to facilitate the rapid retrieval and generation of security plans when the same or similar security demands are input later.

[0224] Tool usage: AI-agent has the ability to use tools. In this application, AI-agent is equipped with various APIs, such as calculators, search tools, database query tools, etc. By calling these tool APIs, AI-agent can create various intelligent applications, which can facilitate human-computer interaction and solve practical demand guarantee problems.

[0225] In summary, a method for transforming support forces based on a large model, first, constructs a professional field dataset by acquiring local resource force support data; then, uses this dataset to pre-train the ChatGLM-6B model to generate a pre-trained model. Next, the model is further optimized through supervised adjustment and RLHF parameter fine-tuning. The optimized model will be deployed on the edge and integrated with LangChain to form an RLHF interaction model. Finally, combined with AI-agent, the RLHF interaction model can decompose and process different categories of support demand tasks to achieve effective transformation of support forces and optimal allocation of resources.

[0226] This method uses ChatGLM-6B as the basic training model. Since the original training data of ChatGLM-6B is bilingual in Chinese and English, the ratio of Chinese to English is 1:1, and the vocabulary covers most of the common Chinese characters and common Chinese words, the Chinese encoding efficiency is high. Therefore, this application uses ChatGLM-6B as the training model. Combined with the pre-training, supervised fine-tuning and RLHF fine-tuning of the large model, the model's language understanding and generation capabilities in specific professional fields are significantly improved. Through edge deployment and LangChain integration, the model's rapid response and efficient interaction are achieved. The introduction of AI-agent enables the system to handle complex security demand tasks more intelligently, improving the efficiency and accuracy of task processing. Overall, this method provides strong technical support for the agile empowerment and precise release of local resource forces, and enhances local security capabilities in complex scenarios.

[0227] In a second aspect, the present application provides a large model-based support force conversion system, which is applied to the aforementioned large model-based support force conversion method, and the system includes:

[0228] The acquisition unit is used to obtain local resource force guarantee data and obtain professional field data sets;

[0229] A training unit, used to pre-train the ChatGLM-6B model using the professional field data set to obtain a pre-trained model;

[0230] A supervision unit is used to supervise and adjust the pre-trained model according to preset prompts and answer samples to obtain a supervised training model;

[0231] A fine-tuning unit, used for fine-tuning the RLHF parameters of the supervised training model to obtain the RLHF training model;

[0232] The interaction unit is used to deploy the RLHF training model on the edge and integrate it with LangChain to obtain the RLHF interaction model;

[0233] The result unit is used to utilize the AI-agent in combination with the RLHF interaction model to perform the guarantee force conversion.

[0234] In a third aspect, the present application provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the aforementioned large-model-based support force conversion method when executing the computer program.

[0235] In a fourth aspect, the present application provides a computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the aforementioned large-model-based support force conversion method.

[0236] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0237] The various embodiments in the present disclosure are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0238] The protection scope of the present disclosure is not limited to the above-mentioned embodiments. Obviously, those skilled in the art can make various changes and modifications to the present disclosure without departing from the scope and spirit of the present disclosure. If these changes and modifications fall within the scope of the claims of the present disclosure and their equivalents, the intention of the present disclosure also includes these changes and modifications.

Claims

1. A method for converting support forces based on a large model, characterized in that: Methods include: Obtain local resource support data and obtain professional field data sets; Using the professional field dataset, pre-training the ChatGLM-6B model to obtain a pre-trained model; According to the preset prompts and answer samples, the pre-trained model is supervised and adjusted to obtain a supervised training model; Fine-tuning the RLHF parameters of the supervised training model to obtain the RLHF training model; Deploy the RLHF training model on the edge and integrate it with LangChain to obtain the RLHF interaction model; The AI-agent is used in combination with the RLHF interaction model to carry out the guarantee force conversion.

2. The method for converting support forces based on a large model according to claim 1 is characterized in that: The ChatGLM-6B model is pre-trained using the professional field dataset to obtain the pre-trained model, including: Performing structured division on the data in the professional field data set to obtain structured data, semi-structured data and unstructured data; Converting the semi-structured data and combining it with the structured data to obtain a first initial corpus; Using regular expressions and natural language processing tools, the unstructured data is processed to obtain a second initial corpus; According to a preset division ratio, the target corpus is divided into two parts to obtain a training set and a validation set, wherein the target corpus includes the first initial corpus and the second initial corpus, and the first initial corpus and the second initial corpus are both texts; Segmenting the data in the training set and the validation set respectively to obtain paragraphs or sentences corresponding to the text; Using a word segmenter, segment the paragraph or sentence corresponding to the text to obtain words or subwords corresponding to the paragraph or sentence corresponding to the text; Mapping the word or subword to an integer code to obtain an integer value corresponding to the word or subword; Input the integer value corresponding to the training set into the initial word segmentation model to obtain an intermediate word segmentation model; Inputting the integer value corresponding to the validation set into the intermediate word segmentation model, and adjusting the parameters of the intermediate word segmentation model according to the result output by the intermediate word segmentation model to obtain a target word segmentation model; The vocabulary in the target word segmentation model is combined with the ChatGLM-6B original word segmentation model to obtain a new word segmentation model; The ChatGLM-6B model is trained using the new word segmentation model to obtain a pre-trained model.

3. The method for converting support forces based on a large model according to claim 1 is characterized in that: The step of performing supervised adjustment on the pre-trained model according to the preset prompts and answer samples to obtain the supervised training model includes: Construct and obtain a number of preset samples with prompts and answers to obtain a preset training set; In the preset training set, the prompt and answer of each preset sample are associated to obtain a continuous input sequence; Using the markers, segmenting the prompt and the answer in the preset sample to obtain a prompt part and an answer part; Using an autoregressive objective function, the loss of the prompt part corresponding to the user prompt is reset to zero to obtain the back propagation of the answer part; According to the back propagation of the answer part, the parameters of the adjustable model part are adjusted, and the parameters of the frozen model part remain unchanged, so as to obtain a supervised training model, wherein the adjustable model part and the frozen model part constitute the pre-training model.

4. The method for converting support forces based on a large model according to claim 1 is characterized in that: The step of performing RLHF fine-tuning on the supervised training model to obtain the RLHF training model comprises: Inputting a prompt into the supervised training model to obtain a plurality of candidate answers output by the supervised training model; Obtaining ranking and scoring of the plurality of candidate answers output by the supervised training model to obtain a score for each candidate answer; Extracting features of the candidate answers to obtain features of the candidate answers; The input prompt and the candidate answer features are used as input, and the scores of the candidate answers are used as labels to form a reward sample, and a reward training set is obtained based on a plurality of the reward samples; Using the reward training set, training an initial reward model to obtain a target reward model; The target reward model is used to fine-tune the RLHF parameters of the supervised training model to obtain the RLHF training model.

5. The method for converting support forces based on a large model according to claim 1 is characterized in that: The step of edge deploying the RLHF training model and integrating it with LangChain to obtain the RLHF interaction model includes: Establishing an index for the professional domain dataset, wherein the index represents the division of text in the professional domain dataset into blocks to form document block vectors; Obtaining an input prompt from a user, and converting the input prompt into a vector to obtain an input vector; Compare the input vector with the document block vectors in the professional domain data set to obtain K document blocks with the highest similarity, wherein the K document blocks with the highest similarity serve as enhanced context information for the input prompt; The K document blocks with the highest similarity are merged with the input prompt to obtain the prompt input into the RLHF training model, thereby forming an edge deployment of the RLHF training model.

6. The method for converting support forces based on a large model according to claim 1 is characterized in that: The step of edge deploying the RLHF training model and integrating it with LangChain to obtain the RLHF interaction model includes: Importing the LangChain library into the interactive environment formed by the user and the RLHF training model; Initialize LangChain and create a LangChain instance connected to the RLHF training model to obtain the RLHF interaction model.

7. The method for converting support forces based on a large model according to claim 1 is characterized in that: The step of utilizing the AI-agent in combination with the RLHF interaction model to perform the safeguard force conversion comprises: Acquire a security requirement task, wherein the security requirement task includes security tasks of different categories; According to the guarantee category, the guarantee demand task is decomposed by using AI-agent to obtain multiple guarantee category subtasks; Inputting a plurality of the security category subtasks into the RLHF interaction model, executing them respectively, and obtaining output results of each of the security category subtasks; According to the output results of the plurality of guarantee category subtasks, a guarantee force conversion is obtained to solve the actual demand guarantee prompts.

8. A large-scale model-based support force conversion system, characterized in that: The large model-based support force conversion method applied to any one of claims 1-7, the system comprising: The acquisition unit is used to obtain local resource force guarantee data and obtain professional field data sets; A training unit, used to pre-train the ChatGLM-6B model using the professional field data set to obtain a pre-trained model; A supervision unit is used to supervise and adjust the pre-trained model according to preset prompts and answer samples to obtain a supervised training model; A fine-tuning unit, used for fine-tuning the RLHF parameters of the supervised training model to obtain the RLHF training model; The interaction unit is used to deploy the RLHF training model on the edge and integrate it with LangChain to obtain the RLHF interaction model; The result unit is used to utilize the AI-agent in combination with the RLHF interaction model to perform the guarantee force conversion.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the large model-based support force conversion method described in any one of claims 1 to 7 are implemented.

10. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the large model-based support force conversion method described in any one of claims 1 to 7 are implemented.

Images