Large model prompt instruction generation method, system and equipment and storage medium

Abstract

The invention provides a large model prompt instruction generation method, system and device and a storage medium. The method comprises the steps of obtaining a text sentence pattern corresponding to text data; corresponding instruction information is determined in a prompt project library of the corresponding target model based on the text sentence pattern, prompt information is determined according to the instruction information, and the prompt information is used for guiding the target model to understand semantics of the instruction information; determining corresponding constraint information based on the text sentence pattern, wherein the constraint information is used for guiding a target model to generate response information meeting a preset requirement; and combining the prompt information, the text sentence pattern and the constraint information, and determining a prompt instruction corresponding to a target model. According to the method, the corresponding instruction information is determined in the prompt project library of the corresponding target model, and the prompt information is determined according to the instruction information, so that the semantics of the instruction information can be further refined, a clearer and more specific guidance is provided for the large model, and the large model is more matched with the processing and response of the large model.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more specifically to a method, system, device, and storage medium for generating large model prompt instructions. Background Technology

[0002] With the rapid development of artificial intelligence technology, large model prompting generation technology plays a crucial role, directly affecting the understanding and processing effectiveness of large models on text data. However, the shortcomings and deficiencies of existing large model prompting technologies limit the performance and effectiveness of large models in handling practical tasks.

[0003] First, most existing methods for generating prompts for large models rely on fixed templates or rules, which results in large models lacking sufficient contextual understanding and adaptability when receiving and processing text data, thus affecting the accurate parsing and response of large models to input information.

[0004] Secondly, existing methods for generating prompts for large models often face problems such as high computational resource consumption and slow processing speed when dealing with large-scale text data. In particular, for datasets containing complex sentence structures and long texts, existing methods struggle to achieve efficient prompt generation while ensuring accuracy. Summary of the Invention

[0005] One of the objectives of this invention is to provide a method for generating prompts for large models, in order to solve the technical problem that the processing and response of large models to text data are relatively fixed in the prior art, resulting in a low degree of matching between large models and input data in terms of understanding and response.

[0006] One of the objectives of this invention is to provide a large model prompt instruction generation system.

[0007] To achieve one of the aforementioned objectives, this invention provides a method for generating prompt instructions for a large model, comprising: acquiring a text sentence corresponding to text data; determining corresponding instruction information in a prompt engineering library of a corresponding target model based on the text sentence, and determining prompt information based on the instruction information, wherein the prompt information is used to guide the target model to understand the semantics of the instruction information; determining corresponding constraint information based on the text sentence, wherein the constraint information is used to guide the target model to generate response information that meets preset requirements; and combining the prompt information, the text sentence, and the constraint information to determine a prompt instruction corresponding to the target model.

[0008] As a further improvement of one embodiment of the present invention, the step of determining the prompt information based on the instruction information includes: performing coarse-grained extraction and / or fine-grained extraction on the instruction information based on the text type corresponding to the instruction information to obtain the prompt information; wherein, the coarse-grained extraction is used to retrieve and extract the overall structure of the instruction information, and the fine-grained extraction is used to retrieve and extract the semantic information of the instruction information.

[0009] As a further improvement of one embodiment of the present invention, the step of performing coarse-grained extraction and / or fine-grained extraction on the instruction information based on the text type corresponding to the instruction information includes: determining whether the text type is long text; if so, extracting key features of the instruction information based on data distillation, inputting the key features into a context recognition model to obtain text features with context information, and performing coarse-grained extraction on the text features.

[0010] As a further improvement of one embodiment of the present invention, after determining whether the text type is long text, the method further includes: if not, parsing and extracting key sentences in the instruction information, and performing fine-grained extraction operations on the key sentences; and / or, parsing and extracting the token field in the instruction information, and performing fine-grained extraction operations on the token field.

[0011] As a further improvement of one embodiment of the present invention, the method further includes: constructing a multimodal large model, and improving the multimodal large model to obtain an improved multimodal large model; acquiring input data, and inputting the input data and the model prompt information into the improved multimodal large model for processing to obtain corresponding output data.

[0012] As a further improvement of one embodiment of the present invention, the construction of a multimodal large model and the improvement of the multimodal large model to obtain an improved multimodal large model include: constructing an initial multimodal large model, the initial multimodal large model including an embedding layer, an activation layer and a feedforward neural network module; pruning the embedding layer and the activation layer to obtain an improved embedding layer and an improved activation layer; improving the feedforward neural network module and setting the improved feedforward neural network module after the improved activation layer to obtain the improved multimodal large model.

[0013] As a further improvement of one embodiment of the present invention, the initial multimodal large model further includes an attention mechanism module, which includes a multi-head attention layer, a residual connection block and a normalization layer; the multi-head attention layer is disposed on the activation layer, the residual connection block is disposed between the multi-head attention layer and the normalization layer, and the normalization layer is connected to the feedforward neural network module.

[0014] As a further improvement of one embodiment of the present invention, the feedforward neural network module includes several feedforward neural network expert models, and the improvement of the feedforward neural network module includes: constructing a gate control operation model with a forget gate, and using the gate control operation model to adjust the weights of several feedforward neural network expert models.

[0015] As a further improvement of one embodiment of the present invention, the feedforward neural network module includes several feedforward neural network expert models, and the improvement of the feedforward neural network module includes: based on dynamic token routing, distributing the model prompt information input to the multimodal large model to the corresponding feedforward neural network expert model.

[0016] As a further improvement of one embodiment of the present invention, the improved multimodal large model includes an encoding module and a decoding module; the step of inputting the input data and the model prompt information into the improved multimodal large model for processing to obtain corresponding output data includes: inputting the input data into the encoding module to obtain corresponding first output data; determining whether the first output data is text type; if so, performing word segmentation and semantic compression operations on the first output data, and completing the decoding operation based on an autoregressive variable decoder to obtain corresponding second output data.

[0017] As a further improvement of one embodiment of the present invention, after determining whether the first output data is text type, the method further includes: if not, fusing the feature information of the first output data based on the cross-attention mechanism to form a fused feature vector, and sequentially inputting the fused feature vector into the convolutional layer, the activation layer and the normalization layer to obtain the corresponding second output data.

[0018] As a further improvement of one embodiment of the present invention, the method further includes: acquiring sample data, and inputting the sample data and the model prompt information into the improved multimodal large model for training to obtain a trained multimodal large model; and constructing a multi-agent collaborative model based on the trained multimodal large model.

[0019] To achieve one of the above-mentioned objectives, the present invention also provides a large model prompt instruction generation system, comprising: an input module for acquiring a text sentence corresponding to text data; a first module for determining corresponding instruction information in a prompt engineering library of the corresponding target model based on the text sentence, and determining prompt information based on the instruction information, wherein the prompt information is used to guide the target model to understand the semantics of the instruction information; a second module for determining corresponding constraint information based on the text sentence, wherein the constraint information is used to guide the target model to generate response information that meets preset requirements; a third module for combining the prompt information, the text sentence, and the constraint information to determine a prompt instruction corresponding to the target model; and an output module for inputting the prompt instruction into a multimodal large model.

[0020] To achieve one of the above-mentioned objectives, the present invention also provides a cooling device, comprising: a memory for storing executable instructions; and a processor for implementing the steps of any of the above-described large model prompt instruction generation methods when running the executable instructions stored in the memory.

[0021] To achieve one of the above-mentioned objectives, the present invention also provides a computer-readable storage medium, comprising: at least one processor; and a memory storing a computer program executable on the processor, wherein the processor executes the program to perform the steps of the large model hint instruction generation method.

[0022] Compared with the prior art, the embodiments of the present invention have at least one of the following beneficial effects:

[0023] This invention employs a large-model prompt instruction generation method. By introducing and extracting instruction information closely related to text sentence structure, prompt information is obtained. This prompt information can further refine the semantics of the instruction information, providing more explicit and specific guidance for the target model, making it more compatible with the target model's processing and response. Moreover, it allows the target model to combine prompt information, text sentence structure, and constraint information to determine prompt instructions. The target model can generate response information that meets user expectations, improving flexibility and adaptability when processing diverse text data. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the structure of a large model prompt instruction generation system according to one embodiment of the present invention.

[0025] Figure 2 This is a schematic diagram illustrating the steps of a method for generating large model prompt instructions in one embodiment of the present invention.

[0026] Figure 3 This is a schematic diagram of the process of multimodal data acquisition and text data generation in one embodiment of the present invention.

[0027] Figure 4 This is a schematic diagram of the process for generating large model prompt instructions in one embodiment of the present invention.

[0028] Figure 5 This is a schematic diagram of step S2 in one embodiment of the present invention.

[0029] Figure 6 This is a schematic diagram of the process for generating prompt information in one embodiment of the present invention.

[0030] Figure 7 This is a schematic diagram of step M1 in a specific embodiment of the present invention.

[0031] Figure 8 This is a schematic diagram of step M2 in a specific embodiment of an embodiment of the present invention.

[0032] Figure 9 This is a schematic diagram of the structure of a multimodal large model in one embodiment of the present invention.

[0033] Figure 10 This is a schematic diagram of the steps of a method for generating large model prompt instructions in one embodiment of the present invention.

[0034] Figure 11 This is a schematic diagram of the structure of a multi-agent collaborative system model in one embodiment of the present invention.

[0035] Figure 12 This is a schematic diagram of data transformation in a preferred embodiment of the large model prompt instruction generation method of the present invention.

[0036] Figure 13 A schematic diagram of the structure of an electronic device according to one embodiment of the present invention. Detailed Implementation

[0037] The present invention will now be described in detail with reference to the specific embodiments shown in the accompanying drawings. However, these embodiments do not limit the present invention, and any structural, methodological, or functional modifications made by those skilled in the art based on these embodiments are included within the scope of protection of the present invention.

[0038] It should be noted that the term "comprising" or any other variation thereof is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Furthermore, the terms "first," "second," "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0039] like Figure 1As described above, one embodiment of the present invention provides a large model prompt instruction generation system 100.

[0040] In one embodiment, the large model hint instruction generation system 100 generates model hint instructions according to a large model hint instruction generation method, which are used to guide the model to make the most matching response possible.

[0041] In one specific embodiment, the method for generating large model prompt instructions can be implemented with reference to any of the technical solutions provided below.

[0042] The Large Model Hint Generation System 100 guides large models to perform specific tasks or generate specific content by generating precise, optimized hints. These hints include task instructions that can guide large models to think deeply, innovate, or execute precisely, going beyond simple input or queries.

[0043] The large model prompt instruction generation system 100 includes an input module 11. The input module 11 is used to obtain the text sentence structure corresponding to the text data.

[0044] Text data can refer to any information existing in written form, including but not limited to single statements, phrases, sentences, and paragraphs. The text data can originate from various sources, such as user input and database queries.

[0045] The large model prompt instruction generation system 100 includes a first module 12.

[0046] In one embodiment, the first module 12 is used to determine the corresponding instruction information in the prompt engineering library of the corresponding target model based on the text sentence pattern.

[0047] In one specific embodiment, the matching prompt number is determined based on the text sentence query prompt library; and the corresponding instruction information is determined based on the prompt number.

[0048] The aforementioned prompting library is a collection of optimized and designed prompts designed to help users interact more effectively with the target model and obtain relevant and useful responses.

[0049] In one specific embodiment, the target model can be any large model.

[0050] In one embodiment, the first module 12 is used to determine prompt information based on the instruction information, and the prompt information is used to guide the target model to understand the semantics of the instruction information.

[0051] Specifically, useful prompts are extracted based on the text type of the instruction information. If the instruction information is long text, such as a PDF or Word document instruction manual, useful contextual prompts are extracted based on the context of the instruction information; if the instruction information is short text, such as a sentence, useful prompts are extracted based on fields or sentences.

[0052] The large model prompt instruction generation system 100 includes a second module 13.

[0053] In one embodiment, the second module 13 is used to determine the corresponding constraint information based on the text sentence, and the constraint information is used to guide the target model to generate response information that meets preset requirements.

[0054] Specifically, the constraint information includes prompt parameters. These prompt parameters include, but are not limited to, at least one of format parameters, theme parameters, and style parameters.

[0055] In one embodiment, the second module 13 is used to combine the prompt information, the text sentence, and the constraint information to determine a prompt instruction corresponding to the target model.

[0056] The large model prompt instruction generation system 100 includes an output module 14. The output module 14 is used to input the prompt instructions into the multimodal large model to guide the multimodal large model to generate response information that better matches the input data.

[0057] like Figure 2 As shown, one embodiment of the present invention provides a method for generating large model prompt instructions.

[0058] The method for generating large model prompts is applied to a large model prompt generation system.

[0059] In one embodiment, the large model prompt instruction generation system can be as follows: Figure 1 The configuration described above is applied, and the corresponding technical solutions are referenced in the generation method provided by this invention. However, the large model prompt instruction generation system used in the large model prompt instruction generation method provided by this invention is not limited to this configuration structure.

[0060] like Figure 2 As shown, a method for generating large model prompts according to one embodiment of the present invention includes the following steps.

[0061] Step S1: Obtain the text sentence structure corresponding to the text data;

[0062] Step S2: Determine the corresponding instruction information based on the text sentence structure, extract the instruction information, and determine the prompt information. The prompt information is used to guide the model to understand the semantics of the instruction information.

[0063] Step S3: Determine the corresponding constraint information based on the text sentence structure. The constraint information is used to guide the model to generate response information that meets the preset requirements.

[0064] Step S4: Combine the prompt information, the text sentence, and the constraint information to determine the model prompt instruction.

[0065] Thus, by introducing and extracting instruction information closely related to the text sentence structure, prompt information can be obtained. This prompt information can further refine the semantics of the instruction information, providing more explicit and specific guidance for the target model, making it more compatible with the target model's processing and response. Moreover, it allows the target model to combine prompt information, text sentence structure, and constraint information to determine prompt instructions. The target model can generate response information that meets user expectations, improving its flexibility and adaptability when processing diverse text data.

[0066] The text data includes multimodal data, including multi-source data (i.e., data from diverse sources), or heterogeneous data (i.e., data with different storage structure types), or multi-source and heterogeneous data. This invention does not impose specific limitations on this.

[0067] In one embodiment, the text data includes at least one of voice text data, video text data, and text data.

[0068] Thus, by obtaining multimodal data, on the one hand, multiple sources can ensure the diversity of data sources, making the obtained text data more complete, comprehensive, and reliable; on the other hand, heterogeneity can connect different kinds of information together based on differences in data structure, enriching the text content of the text data from the aspects of multiple structures and types of data.

[0069] The voice-text data may refer to text content transcribed from real-time or offline voice data; the video-text data may refer to text content transcribed from real-time or offline video data; the text data may include real-time and offline text data, specifically text data related to user preferences, hobbies, interests, or comments regarding ingredients.

[0070] Understandably, to enrich the training dataset or corpus data and facilitate accurate construction of the natural language model, the text data may also include historical speech, video, and text data; this invention does not impose specific limitations on this. Similarly, these can be converted into corresponding text data, and the methods used can be referred to the description of the specific embodiments below, which will not be repeated here.

[0071] like Figure 3As shown, in one specific embodiment, multimodal data is collected from multiple channels, and the multimodal data is processed separately to convert it into valid text data. The valid text data is then aggregated and filtered to obtain valid text sentences. The valid text data includes at least one of the following: speech-to-text data (first text data), video / image-to-text data (second text data), and plain text data.

[0072] Specifically, real-time voice data can be collected from users through voice acquisition devices such as microphones and microphone arrays installed inside the smart refrigerator; real-time images of users can also be collected through video acquisition devices such as mobile phones and / or cameras installed inside the refrigerator, or images can be obtained based on public accounts or mini programs; audio and video data can also be collected through video acquisition tools and web scraping; similarly, real-time text data can also be obtained through mobile phones, tablets, and applications such as imaging, mini programs, public accounts, or the Web, HTTP (Hypertext Transfer Protocol), API (Application Programming Interface), or SDK (Software Development Kit).

[0073] The collected data undergoes preprocessing. Specifically, the speech is segmented according to a specified length (time period or number of samples), completing frame segmentation and windowing processing. Further, pre-emphasis processing can be performed before frame segmentation to emphasize the high-frequency components of the speech, eliminating the influence of lip radiation during phonation, thus compensating for the high-frequency components suppressed by the vocal system and highlighting high-frequency formants. After windowing, steps such as filtering audio noise and enhancing human voice can be performed to strengthen the real-time speech data, extract feature parameters, and ensure the real-time speech data meets the input requirements of subsequent neural network models.

[0074] In one specific embodiment, irrelevant data, duplicate data, outliers, and missing values ​​are removed from the text dataset to clean the text data, thereby making the text data meet the input requirements of the subsequent neural network model.

[0075] In one specific embodiment, the audio-video separation script can be written using Python, or a third-party audio-video separation tool can be used to perform separation operations on data that simultaneously contains video and audio data to obtain valid voice and video data.

[0076] In one specific embodiment, the real-time video data can be cropped by writing a script or using a third-party video cropping tool. Specifically, the video data is loaded and the video information is read, the video is decoded according to the video information, and multi-frame image data corresponding to the real-time video data is obtained based on the width and height of a single frame image.

[0077] Continue to refer to Figure 3 As shown, in one specific embodiment, it is determined whether the input data contains speech data; if so, the speech data is input into a convolutional enhanced neural network model to determine the first text data corresponding to the speech data.

[0078] In one specific embodiment, the convolution-enhanced neural network model includes a convolutional neural network and a Transformer model or a Transformer Block model. Specifically, convolutional layers are introduced into the Transformer architecture to enhance the model's ability to capture local features. This combination allows the model to retain the advantages of Transformer in modeling global interactive speech features while more effectively utilizing local feature information, thereby improving performance in tasks such as speech recognition.

[0079] In one specific embodiment, it is determined whether the input data contains video data; if so, the video data is input into the distillation-diffusion model to determine the second text data corresponding to the video data.

[0080] Specifically, the distillation-diffusion model may include two steps. In the first step, a continuous-time student model is introduced to match the combined output of two teacher diffusion models to obtain a distillation model. In the second step, the distillation model trained in the first step is gradually transformed into a model with fewer steps until the teacher model is distilled to half the number of steps of the student model.

[0081] Continue to refer to Figure 3 As shown, in another specific embodiment, it is determined whether the input data contains video data; if so, the video data is input into the multimodal pre-training model to determine the second text data corresponding to the video data. The multimodal pre-training model includes a Contrastive Language-Image Pre-training (CLIP) model.

[0082] Specifically, the multimodal pre-trained model includes an image encoder and a text encoder. The image encoder, which can be a convolutional neural network, converts images into feature vectors; the text encoder converts text into feature vectors. Images and text are converted into feature vectors by their respective encoders. These feature vectors represent the semantic representations of the images and text. A contrastive loss function is used to ensure that the feature vectors of matching image-text pairs are as close as possible, while the feature vectors of mismatched image-text pairs are as far apart as possible, thus achieving correlation between the image and text features in the same spatial domain. In other words, relevant text descriptions are generated based on the image content.

[0083] like Figure 4 As shown, in one specific embodiment, the system queries the prompt engineering library corresponding to the target model based on valid text sentence patterns to determine the matching prompt number; based on the prompt number, it determines the corresponding instruction information and constraint information; it extracts the instruction information to determine the prompt information; and it combines the prompt information, text sentence patterns, and constraint information to determine the prompt instruction corresponding to the target model.

[0084] The text sentence structure refers to the structure or form of sentences in text data, used to describe the order of words, grammatical relationships, and semantic organization within the sentence. The text sentence structure includes, but is not limited to, sentence type, sentence components, grammatical relationships, and semantic expression.

[0085] The prompt number is a unique identifier within the prompt knowledge base that identifies the instruction information. The instruction information serves as a strategy, method, or guidance to help the large model better understand the input text data and generate a response that meets the requirements. The constraint information limits the model's degrees of freedom in generating response information, ensuring that the output response information meets preset standards or requirements across multiple dimensions.

[0086] In one embodiment, step S2, "determining the prompt information based on the instruction information," may include the following steps.

[0087] Step S2': Based on the text type corresponding to the instruction information, perform coarse-grained extraction and / or fine-grained extraction on the instruction information to obtain the prompt information.

[0088] In this way, by combining coarse-grained and fine-grained extraction, different task requirements can be flexibly addressed. This avoids deviations in task execution due to overly simplistic information, and also prevents reduced processing efficiency due to overly complex information.

[0089] The coarse-grained extraction method is used to retrieve and extract the overall structure of the instruction information, focusing on the main components of the text and the relationships between them, rather than paying too much attention to details. During coarse-grained extraction, some non-critical details or specific information may be ignored to preserve the core framework and key points of the instruction information.

[0090] The fine-grained extraction is used to retrieve and extract the semantic information of the instruction, focusing on the specific meaning of each word, phrase, or sentence in the text and the logical relationships between them. During the fine-grained extraction process, key details or strong textual content and subtle differences are preserved as much as possible to ensure that the extracted text accurately conveys the meaning of the original information.

[0091] The term "strong text" refers to textual content that is of particular importance, prominence, or emphasis. These sections may contain key information, important details, emotionally charged expressions, or points that the author wants the reader to pay special attention to.

[0092] like Figure 5 As shown, in one specific embodiment, step S2' may include, but is not limited to, the following steps.

[0093] Step S21: Determine whether the text type is a long text;

[0094] If so, proceed to step S22A, extract key features of the instruction information based on data distillation, input the key features into the context recognition model to obtain text features with context information, and perform coarse-grained extraction operation on the text features.

[0095] If not, proceed to step S22B, parse and extract key sentences from the instruction information, and perform fine-grained extraction operations on the key sentences; and / or, parse and extract the token field from the instruction information, and perform fine-grained extraction operations on the token field.

[0096] Thus, by extracting key features from long texts through data distillation and fusing contextual information using a context recognition model, the core content of long texts can be efficiently summarized, providing strong support for subsequent processing; while for non-long texts, directly parsing and extracting key sentences and / or token fields can quickly locate the most important information points.

[0097] Data distillation is a method of data compression and data refinement. In the process of data distillation, knowledge from a large model dataset or network is extracted and compressed into a smaller dataset or network, while retaining as much key information and features of the original data as possible.

[0098] The aforementioned context recognition model is a model used to process and understand contextual relationships in natural language text. Specifically, it infers the overall meaning and context of the text by analyzing information such as vocabulary, grammar, and semantics, as well as the relationships between these information.

[0099] The token field refers to the independent units, such as words, phrases, or punctuation marks, that are separated from the text. These units are fields or identifiers used to represent specific information or data during data processing or information exchange.

[0100] The extraction operation can be understood as an information compression operation, that is, in the process of information transmission, extracting key information and removing irrelevant details to express the original meaning in a more concise way.

[0101] In one specific embodiment, key features of the instruction information are extracted based on data distillation to form a simplified dataset; annotation is performed on the dataset; the annotated data is input into a context recognition model to obtain text features with context information, and coarse-grained extraction is performed on the text features.

[0102] In this way, by performing annotation operations on the dataset, the necessary supervision information is provided for model training, enabling the model to learn the correct classification or regression results of the data.

[0103] The "long text" in step S21 may refer to text containing multiple paragraphs, longer sentences, and a larger amount of information, but there is no fixed standard for the number of words or characters. The corresponding text type can be determined by comparing the length of the text with a set threshold; the text type includes long text and / or short text.

[0104] In one specific embodiment, it is determined whether the length of the text corresponding to the instruction information exceeds a set threshold; if so, based on data distillation, key features of the instruction information are extracted, and the key features are input into a context recognition model to obtain text features with context information, and a coarse-grained extraction operation is performed on the text features; if not, key sentences in the instruction information are parsed and extracted, and a fine-grained extraction operation is performed on the key sentences; and / or, a token field in the instruction information is parsed and extracted, and a fine-grained extraction operation is performed on the token field.

[0105] like Figure 6As shown, in one specific embodiment, the extraction of instruction information for long text types, such as PDF instruction manuals, can be understood as combining data distillation (training a small model to mimic the behavior of a large model while retaining key information), data annotation (adding additional tags or metadata to the data to enhance its understandability and processability), and contextual information (considering the overall background and contextual relationships of the text) to achieve information compression. This method focuses more on coarse-grained information retrieval and compression from the overall document structure and content.

[0106] For extracting instruction information from non-long text types, depending on the size of the text content, tokens (such as very short text fragments), sentences (such as slightly longer text paragraphs), or tokens and sentences (such as medium-length text) are used as prompts for extraction. This method focuses on detailed analysis and processing of the text, and achieves fine-grained extraction by optimizing the selection and combination of prompts. It aims to preserve key details and semantic information in the text while reducing unnecessary redundancy.

[0107] In one specific embodiment, the "combination" in step S4 can be understood as: matching the extracted prompt information with the requirements of the text sentence structure to ensure that the two are logically consistent; constructing a model prompt information framework based on the matching results, wherein the information framework includes at least one of prompt instructions and sentence structure requirements; and filling the model prompt information framework based on the constraint information to obtain the model prompt instructions.

[0108] Understandably, the prompts after coarse and fine granularity extraction help to better match the processing needs of large models. In order to further improve the matching degree with prompts, the structure of large models can also be improved.

[0109] In one embodiment, the large model prompt instruction generation method of the present invention may further include the following steps.

[0110] Step M1: Construct a multimodal large model and improve the multimodal large model to obtain an improved multimodal large model;

[0111] Step M2: Obtain input data and input the input data and the model prompt information into the improved multimodal large model for processing to obtain the corresponding output data.

[0112] Thus, by inputting the processed prompts into the improved multimodal large model, the matching degree between the two can be improved, and the processing efficiency and accuracy of the response information of the model can be enhanced.

[0113] The multimodal large model is a machine / deep reinforcement learning method based on deep learning that integrates different media data (such as text, images, audio, and video) and achieves more intelligent information processing by learning the relationships between different modalities.

[0114] It should be noted that the multimodal large model includes an encoding module and a decoding module. Therefore, improvements to the multimodal large model include improving at least one of the encoding module and the decoding module. The improvement will be explained from the perspective of the encoding module.

[0115] like Figure 7 As shown, in a specific embodiment, step M1 of the present invention may specifically include the following steps.

[0116] Step M11: Construct an initial multimodal large model, which includes an embedding layer, an activation layer, and a feedforward neural network module;

[0117] Step M12: Prune the embedding layer and the activation layer to obtain the improved embedding layer and the improved activation layer;

[0118] Step M13: Improve the feedforward neural network module and set the improved feedforward neural network module after the improved activation layer to obtain the improved multimodal large model.

[0119] Thus, pruning can remove redundant or unnecessary parameters in the embedding and activation layers, reducing the complexity and computational cost of the model.

[0120] Specifically, residual and normalization-based pruning operations are performed on neurons in the embedded layer.

[0121] In one specific embodiment, the input data and expected output data of the embedding layer are obtained; for each neuron in the embedding layer, the difference between its actual output and expected output is calculated; root mean square normalization (RMS) is performed on the input and output data of each neuron (i.e., the root mean square value of the input and output of each neuron is calculated and used as the normalization benchmark); a pruning threshold is determined based on the difference and the RMS normalization result; it is determined whether the RMS normalization result of the neuron is less than the pruning threshold; if so, pruning is performed. After pruning, a validation dataset can be used to validate the pruned model.

[0122] Pruning operations involve removing rows and columns of corresponding neurons from the weight matrix to remove that neuron and its connections with other neurons. The weight matrix refers to the data object used in a multimodal large model to represent the connection strength between neurons in different layers. Specifically, the weight matrix is ​​a two-dimensional matrix whose elements (i.e., weights) represent the connection weights between adjacent neurons in the model.

[0123] Similarly, residual- and normalization-based pruning operations are also performed on neurons in the activation layer. Specific operations can be found in the section on pruning the embedding layer, and will not be elaborated upon here.

[0124] In one specific embodiment, the initial multimodal large model further includes an attention mechanism module, which includes a multi-head attention layer or a multi-head self-attention layer, a residual connection block, and a normalization layer; the multi-head attention layer is disposed on the activation layer, the residual connection block is disposed between the multi-head attention layer and the normalization layer, and the normalization layer is connected to the feedforward neural network module.

[0125] In this way, the multi-head attention mechanism can simultaneously focus on multiple aspects of the input sequence, thereby improving the model's ability to process complex information. In this process, residual connections and normalization are introduced to ensure that the information does not experience gradient weakening or vanishing as the number of network layers increases, thus maintaining the stability of the input data distribution.

[0126] In the above embodiments, the activation layer may include a multi-head attention mechanism. When performing neuron pruning on the activation layer containing the multi-head attention mechanism, special attention needs to be paid to each attention head in the multi-head attention mechanism to ensure that the pruned model still maintains sufficient expressive power. Each attention head can be evaluated individually to determine which parts can be pruned. Furthermore, adding the original input to the output of the multi-head attention mechanism based on residual connections can enhance the model's ability to learn complex features. Additionally, the activation layer can also be connected to the attention mechanism module; no specific restrictions are placed on this.

[0127] In one specific embodiment, the normalization layer may employ root mean square normalization (RMSNorm), which is placed after the multi-head attention layer to normalize the output of the multi-head attention layer. This helps to further stabilize the training process of the model and improve the convergence speed and generalization ability of the model.

[0128] The multi-head attention mechanism uses multiple self-attention mechanisms on an input sequence to obtain multiple sets of attention results, and then concatenates and linearly projects these multiple sets of attention results to obtain the final output result.

[0129] The residual connection block is a neural network model that directly adds the input to the module's output through residual connections (or skip connections); the normalization layer is a neural network layer used to normalize the input features. By adjusting the distribution range of the data, it helps to accelerate the model's training speed, improve the model's stability, and prevent gradient vanishing or exploding.

[0130] In one specific embodiment, the feedforward neural network module includes several feedforward neural network expert models, and the "improvement of the feedforward neural network module" part in step M13 may include the following steps.

[0131] Step M13': Construct a gate control operation model with a forget gate, and use the gate control operation model to adjust the weights of several feedforward neural network expert models.

[0132] In this way, by constructing a gating computation model and dynamically adjusting the weights accordingly, the contribution of different expert models in processing input data can be controlled, thereby achieving fine control over the behavior of expert models.

[0133] A feedforward neural network (FFN) is a neural network architecture in which information propagates unidirectionally within the network, from the input layer to the output layer, without backward propagation. A feedforward neural network module can consist of a single feedforward neural network or a combination of multiple feedforward neural networks.

[0134] Feedforward neural network expert models can be understood as models specifically designed and trained for a particular domain or task, based on feedforward neural networks. For example, in image recognition, feedforward neural network expert models can extract image features by adding structures such as convolutional layers and pooling layers; in natural language processing, they can process sequential data by introducing structures such as recurrent neural networks or long short-term memory networks. Multiple feedforward neural network experts can be combined, with each expert specializing in processing different parts or aspects of the input data, thereby improving the model's processing efficiency and scope.

[0135] Gated computational models with forget gates are used to control the flow of information. They can adaptively control the transmission and filtering of information based on the specificity of the input data and the needs of the model. Through forget gates, the model can more flexibly handle complex relationships in multimodal data, while suppressing unimportant information and retaining key features.

[0136] Specifically, the forget gate in the gating operation model can output a value between 0 and 1 based on the input data and the current state. This value is used as a weighting factor to adjust the weights of the corresponding connections in the expert model. When the output of the forget gate is close to 0, it indicates that the information of the connection should be forgotten or ignored, and the corresponding weight is reduced; when the output is close to 1, it indicates that the information of the connection should be retained, and the corresponding weight remains unchanged or increases slightly. Therefore, the weight in step M13' refers to the connection weights between neurons in each layer of the feedforward neural network expert model, which determines the way and strength of information transmission in the neural network.

[0137] In one specific embodiment, the "improving the feedforward neural network module" part in step M13 may include the following steps.

[0138] Step M13” assigns the model hints input to the multimodal large model to the corresponding feedforward neural network expert model based on dynamic token routing.

[0139] In this way, through dynamic token routing, each expert will receive a load that matches its processing capacity, which helps to dynamically allocate parallel computing resources for multimodal large models and achieve load balancing.

[0140] The dynamic token routing is used to dynamically allocate computational tasks to different feedforward neural network expert models based on the characteristics of the input data and the needs of the model. Specifically, the input data is acquired, segmented, and several tokens are determined. These tokens include, but are not limited to, single data units, pixel blocks, or other forms of data units. Based on these tokens, the input data is allocated to the corresponding expert models. The processing results of all experts are combined to form the final output.

[0141] On the other hand, the decoding module can also be improved. The following is an explanation of the improvements to the decoding module.

[0142] like Figure 8 As shown, the part of step M2 of the present invention that "inputs the input data and the model prompt information into the improved multimodal large model for processing to obtain the corresponding output data" may include the following steps.

[0143] Step M21: Input the input data into the encoding module to obtain the corresponding first output data;

[0144] Step M22: Determine whether the first output data is of text type;

[0145] If so, proceed to step M23A, perform word segmentation and semantic compression on the first output data, and complete the decoding operation based on the autoregressive variable decoder to obtain the corresponding second output data;

[0146] If not, proceed to step M23B, fuse the feature information of the first output data based on the cross-attention mechanism to form a fused feature vector, and input the fused feature vector into the convolutional layer, activation layer and normalization layer in sequence to obtain the corresponding second output data.

[0147] In this way, by performing corresponding processing operations based on the text type of the first output data, the advantages of each processing method can be fully utilized, and the adaptability and comprehensiveness of the model processing can be improved.

[0148] The second output data may include at least one of text data, video data, and audio data.

[0149] Text data has a high degree of sequence and semantics. Segmenting it into words helps break the text down into smaller units, which helps the model better understand the contextual structure of the text. Semantic compression can further extract the core information of the text, remove redundancy, and make subsequent processing more efficient.

[0150] Non-textual data (such as images and audio) has complex structures and features, making it difficult to capture its key information through direct processing. This paper describes a method that processes the first output data using a cross-attention mechanism to determine fusion features (forming a unified feature vector or feature representation from the features themselves and between features). The fusion features are then subjected to convolution, activation function calculations, and normalization layer calculations to obtain the corresponding second output data. This process effectively combines complex features, extracting and transforming the spatial and / or temporal features represented by these data to obtain the corresponding second output data.

[0151] An autoregressive variable decoder is a decoding structure that combines the properties of autoregressive decoders and variational autoencoders. It can be used in conjunction with an encoder-decoder structure, where the encoder encodes the input sequence into a context vector, and the decoder generates the output sequence step by step based on the context vector.

[0152] In one specific embodiment, a multimodal large model is evaluated based on a human feedback reinforcement learning method; the evaluation process may include the calculation of metrics such as accuracy and recall.

[0153] In one specific embodiment, the improved encoding module and the improved decoding module can be combined to form an improved multimodal large model.

[0154] To more clearly illustrate the structure of the improved multimodal large model, for example, such as Figure 9As shown, the encoding and decoding modules are combined. By employing a gating operation model with a forget gate and a multi-head attention mechanism to construct the encoder block, and by using different processing strategies for different data types of the output of the encoding module, the generalization ability of the multimodal large model is improved.

[0155] Continue to refer to Figure 9 As shown, in one embodiment, the multimodal large model further includes a text-to-image module, which is used to convert text information into image information. Specifically, a U-Net deep network model is constructed; initial noise data is obtained; it is determined whether the initial noise data is image data; if so, the initial noise data is input to an image encoder for processing to obtain image semantic features; if not, the initial noise data is input to a text embedding encoder to obtain text semantic features; based on a cross-attention mechanism, the image semantic features and the text semantic features are fused to determine a joint feature representation; the joint feature representation is input to the U-Net deep network model for processing to determine the corresponding image information.

[0156] In this embodiment, the text-to-image module uses the input text embedding and image reference to generate an image that highly matches the text description.

[0157] like Figure 10 As shown in one embodiment, the large model prompt instruction generation method of the present invention may further include the following steps.

[0158] Step N1: Obtain sample data, and input the sample data and the model prompt information into the improved multimodal large model for training to obtain the trained multimodal large model;

[0159] Step N2: Based on the trained multimodal large model, construct a multi-agent collaborative model.

[0160] Thus, based on the perception, understanding, and reasoning capabilities of multimodal large models, intelligent agents can more accurately understand task objectives and environmental information, achieving more efficient collaborative work.

[0161] The term "intelligent agent" refers to an entity possessing intelligence. This entity can be hardware (such as a robot) or software, capable of autonomously perceiving information, making decisions, and executing actions in its environment to achieve specific goals or tasks. The term "multi-agent collaborative model" refers to a system composed of multiple intelligent agents, where these agents collaborate, coordinate, and communicate to jointly complete tasks or achieve goals.

[0162] In one embodiment, the "multi-agent cooperative model" in step N2 can also be an independent "multi-agent cooperative system" or a component of a multi-agent system, namely a "multi-agent cooperative module". The present invention does not impose specific limitations on this.

[0163] like Figure 11 As shown, in one specific embodiment, the multi-agent collaborative model includes a decision-making agent, a planning agent, a behavioral agent, a reflective agent, and a tool agent. The control decision-making agent formulates initial decisions based on business needs and environmental changes; the control planning agent generates specific business processes based on the initial decisions and controls the behavioral agents to execute corresponding operations according to the business processes; during the execution of operations, the control reflective agent monitors and evaluates the business execution process and sends feedback information to the decision-making agent; the control decision-making agent adjusts and optimizes the initial decisions based on the feedback information and determines to update the initial decisions.

[0164] Furthermore, tool agents do not directly participate in the decision-making, planning, or execution of business processes. Instead, they provide other agents with necessary tools, resources, or services, such as data preprocessing, feature processing, and pattern recognition, to enhance the execution capabilities and response speed of other agents.

[0165] Continue to refer to Figure 11 As shown, in one specific embodiment, the multi-agent collaborative model can also establish a communication connection with a knowledge base. Specifically, on the one hand, the multi-agent collaborative model can query the knowledge base to retrieve matching data; on the other hand, the multi-agent collaborative model updates the knowledge base based on decisions and plans, and the knowledge base can be derived from internal knowledge and external knowledge content.

[0166] In one specific embodiment, the trained multimodal large-scale model is deployed. Specifically, a model suitable for the current needs is selected from existing multimodal large-scale models, and a model service application is submitted to the model provider or relevant internal department, including information such as the model name, purpose, and expected usage scale; according to business needs, model engineering plugins are developed or customized, including preprocessing plugins, interface call plugins, etc.; if the model is provided by a third party, the corresponding usage authorization is obtained, and the model call interface is designed and tested for integration; finally, the results returned by the model are processed and displayed.

[0167] The various embodiments, examples, or specific examples provided by this invention can be combined with each other to ultimately form multiple better embodiments.

[0168] For example, Figure 12 This corresponds to the transformation process of the relevant multi-source heterogeneous data or text data involved in the large model prompt instruction generation method when implementing this preferred embodiment. The following will combine... Figure 12 The processing procedure of this preferred embodiment is summarized below.

[0169] First, multi-source heterogeneous data is acquired through multiple channels to obtain various types of data 101. Specifically, the various types of data 101 include text data, voice data, and video data. Text transcription operations are then performed on the various types of data 101, such as preprocessing and feature extraction of voice and / or video data, and transcribing them into corresponding text content to generate corresponding text data 102.

[0170] The transcribed text data 102 is processed to determine the valid text sentence 103. Based on the text sentence 103, the prompt knowledge base is queried to determine the corresponding instruction information 104 and constraint information 105.

[0171] Extract or compress the text corresponding to the instruction information 104 to obtain the corresponding prompt information 106; combine the text sentence 103, constraint information 105 and prompt information 106 to obtain the model prompt instruction 107.

[0172] The model prompt instruction 107 is input to the encoding module of the multimodal large model for processing to obtain the first output data 108, and the first output data 108 is input to the decoding module of the multimodal large model for processing to obtain the second output data 109.

[0173] like Figure 13 As shown, one embodiment of the present invention provides an electronic device 200.

[0174] Electronic device 200 may specifically be a computer device, which may be a terminal device or a server.

[0175] The electronic device 200 includes at least one processor. The large model prompt instruction generation method provided by the present invention can be applied to or implemented by the processor. Specifically, the processor may be a central processing unit (CPU) 21.

[0176] Electronic device 200 includes a memory. The memory is used to store various types of data to support the operation of electronic device 200. Examples of such data include any computer program used to operate on a computer device. The memory may be a read-only memory (ROM) 22, a random access memory (RAM) 23, or other storage portion 28. The storage portion 28 may be located within or outside electronic device 200.

[0177] In one embodiment, when the processor executes a computer program stored in the memory, the step of the large model hint instruction generation method of any technical solution of the present invention is performed.

[0178] In one embodiment, the electronic device 200 includes a central processing unit 21, which can perform various appropriate actions and processes based on a program stored in a read-only memory 22 or a program loaded from a storage section 28 into a random access memory 23. The random access memory 23 also stores various programs and data required for system operation. The central processing unit 21, the read-only memory 22, and the random access memory 23 are interconnected via a bus 24. An input / output interface (I / O interface) 25 is also connected to the bus 24.

[0179] The following components are connected to the input / output interface 25: an input section 26 including a keyboard, mouse, etc.; an output section 27 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 28 including a hard disk, etc.; and a communication section 29 including a network interface card such as a local area network card, modem, etc. The communication section 29 performs communication processing via a network such as the Internet. A drive 210 is also connected to the input / output interface 25 as needed. A removable medium 211, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 210 as needed so that computer programs read from it can be installed into the storage section 28 as needed.

[0180] One embodiment of the present invention provides a computer-readable storage medium.

[0181] In one embodiment, a computer-readable storage medium stores a computer program executed by the processor mentioned above, or a large model hint instruction generation method from any of the preceding technical solutions.

[0182] When the processor executes a computer program, it can perform the description of the large model hint instruction generation method in any of the preceding technical solutions; therefore, it will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated.

[0183] The computer-readable storage medium may include: flash drive, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc.

[0184] In summary, this invention provides a method, system, device, and storage medium for generating prompts for large-scale models. By introducing and extracting instruction information closely related to text sentence structure, prompt information is obtained. This prompt information further refines the semantics of the instruction information, providing more explicit and specific guidance to the target model, making it more compatible with the target model's processing and response. Furthermore, it allows the target model to combine prompt information, text sentence structure, and constraint information to determine the prompt instructions, enabling the target model to generate response information that meets user expectations, improving flexibility and adaptability when processing diverse text data. In addition, extracting instruction information can reduce data transmission and processing time overhead while maintaining information integrity. Thus, the target model can respond more quickly after receiving prompt instructions, improving overall processing efficiency.

[0185] It should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This way of describing the specification is only for clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

[0186] The detailed descriptions listed above are merely specific descriptions of feasible embodiments of the present invention, and are not intended to limit the scope of protection of the present invention. All equivalent embodiments or modifications made without departing from the spirit of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for generating prompts for large models, characterized in that, include: Retrieve the text sentence structure corresponding to the text data; Based on the text sentence structure, the corresponding instruction information is determined in the prompting engineering library of the corresponding target model, and prompt information is determined according to the instruction information. The prompt information is used to guide the target model to understand the semantics of the instruction information. Based on the text sentence structure, the corresponding constraint information is determined, and the constraint information is used to guide the target model to generate response information that meets the preset requirements; By combining the prompt information, the text format, and the constraint information, a prompt instruction corresponding to the target model is determined.

2. The method for generating large model prompt instructions according to claim 1, characterized in that, The step of determining the prompt information based on the instruction information includes: Based on the text type corresponding to the instruction information, coarse-grained extraction and / or fine-grained extraction are performed on the instruction information to obtain the prompt information; wherein, the coarse-grained extraction is used to retrieve and extract the overall structure of the instruction information, and the fine-grained extraction is used to retrieve and extract the semantic information of the instruction information.

3. The method for generating large model prompt instructions according to claim 2, characterized in that, The step of performing coarse-grained extraction and / or fine-grained extraction on the instruction information based on the text type corresponding to the instruction information includes: Determine whether the text type is a long text; If so, based on data distillation, the key features of the instruction information are extracted, the key features are input into the context recognition model to obtain text features with context information, and coarse-grained extraction is performed on the text features.

4. The method for generating large model prompt instructions according to claim 3, characterized in that, After determining whether the text type is long text, the method further includes: If not, then parse and extract the key sentences in the instruction information, and perform fine-grained extraction operations on the key sentences; and / or, parse and extract the token field in the instruction information, and perform fine-grained extraction operations on the token field.

5. The method for generating large model prompt instructions according to claim 1, characterized in that, The method further includes: A multimodal large model is constructed, and the multimodal large model is improved to obtain an improved multimodal large model; The input data is acquired, and the input data and the model prompt information are input into the improved multimodal large model for processing to obtain the corresponding output data.

6. The method for generating large model prompt instructions according to claim 5, characterized in that, The construction of the multimodal large model and the improvement of the multimodal large model to obtain the improved multimodal large model include: Construct an initial multimodal large model, which includes an embedding layer, an activation layer, and a feedforward neural network module; The embedding layer and the activation layer are pruned to obtain an improved embedding layer and an improved activation layer; The feedforward neural network module is improved, and the improved feedforward neural network module is set after the improved activation layer to obtain the improved multimodal large model.

7. The method for generating large model prompt instructions according to claim 6, characterized in that, The initial multimodal large model also includes an attention mechanism module, which includes a multi-head attention layer, a residual connection block, and a normalization layer. The multi-head attention layer is set in the activation layer, the residual connection block is set between the multi-head attention layer and the normalization layer, and the normalization layer is connected to the feedforward neural network module.

8. The method for generating large model prompt instructions according to claim 6, characterized in that, The feedforward neural network module includes several feedforward neural network expert models, and the improvement of the feedforward neural network module includes: Construct a gate control operation model with a forget gate or a forgetting gate, and use the gate control operation model to adjust the weights of several feedforward neural network expert models.

9. The method for generating large model prompt instructions according to claim 6, characterized in that, The feedforward neural network module includes several feedforward neural network expert models, and the improvement of the feedforward neural network module includes: Based on dynamic token routing, model hints input to a multimodal large model are distributed to the corresponding feedforward neural network expert model.

10. The method for generating large model prompt instructions according to claim 5, characterized in that, The improved multimodal large model includes an encoding module and a decoding module; The step of inputting the input data and the model prompt information into the improved multimodal large model for processing to obtain corresponding output data includes: The input data is input to the encoding module to obtain the corresponding first output data; Determine whether the first output data is of text type; If so, then the first output data is segmented and semantically compressed, and the decoding operation is completed based on the autoregressive variable decoder to obtain the corresponding second output data.

11. The method for generating large model prompt instructions according to claim 10, characterized in that, After determining whether the first output data is text type, the method further includes: If not, the feature information of the first output data is fused based on the cross-attention mechanism to form a fused feature vector, and the fused feature vector is sequentially input into the convolutional layer, activation layer and normalization layer to obtain the corresponding second output data.

12. The method for generating large model prompt instructions according to claim 5, characterized in that, The method further includes: Acquire sample data, and input the sample data and the model prompt information into the improved multimodal large model for training to obtain a trained multimodal large model; Based on the trained multimodal large model, a multi-agent collaborative model is constructed.

13. A system for generating prompting instructions for large models, characterized in that, include: The input module is used to obtain the text sentence structure corresponding to the text data; The first module is used to determine the corresponding instruction information in the prompting engineering library of the corresponding target model based on the text sentence pattern, and to determine the prompt information according to the instruction information. The prompt information is used to guide the target model to understand the semantics of the instruction information. The second module is used to determine the corresponding constraint information based on the text sentence structure. The constraint information is used to guide the target model to generate response information that meets the preset requirements. Used to combine the prompt information, the text sentence, and the constraint information to determine the prompt instruction corresponding to the target model; The output module is used to input the prompting instructions into the multimodal large model.

14. A refrigeration device, comprising: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the steps of the large model hint instruction generation method according to any one of claims 1 to 12.

15. A computer-readable storage medium comprising: At least one processor; A memory storing a computer program executable on the processor, characterized in that the processor executes the program by performing the steps of the large model hint instruction generation method as described in claims 1 to 12.

Images