Training method of cue word quality evaluation model and quality evaluation method

Abstract

The invention discloses a training method of a cue word quality evaluation model and a quality evaluation method.The training method comprises the steps that a cue word training text is obtained, text sequence word segmentation is conducted on the cue word training text, and a semantic accuracy sequence, an expression integrity sequence and a logic continuity sequence of the cue word training text are obtained; according to the semantic accuracy sequence, the expression integrity sequence and the logic continuity sequence, performing multi-dimensional parallel coding fusion processing to obtain a target cue word representation vector; performing double-branch prediction fusion on the target cue word representation vector to obtain fused prediction data; and according to the fusion prediction data, performing parameter updating on the initialized cue word quality evaluation model to obtain a trained cue word quality evaluation model. According to the training method, a cue word quality evaluation model can be provided, and the model can be beneficial to improving the interpretability and accuracy of cue word quality evaluation. The invention relates to the technical field of natural language processing.

Description

A training method and quality assessment method for a prompt word quality assessment model Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a training method and a quality assessment method for a prompt word quality assessment model. Background Technology

[0002] In recent years, with the rapid development of natural language processing technology, its application in text generation, intelligent question answering, code-assisted creation and other fields has become increasingly widespread. As the core interactive carrier connecting human intentions and AI models, the quality of prompt words directly determines the task output effect of AI. Therefore, prompt word quality assessment has gradually become one of the key focuses of relevant practitioners.

[0003] Currently, related technologies typically assess the quality of prompt words based on pre-trained language models. However, since this approach often directly adopts common text classification paradigms, it is difficult to analyze the key factors affecting the quality of prompt words, resulting in unsatisfactory interpretability and accuracy of prompt word quality assessment.

[0004] Therefore, the problems with the relevant technologies still need to be solved and optimized. Summary of the Invention

[0005] The purpose of this invention is to at least partially solve one of the technical problems existing in the related art.

[0006] Therefore, one objective of this invention is to provide a training method and a quality assessment method for a prompt word quality assessment model. The training method can provide a prompt word quality assessment model that can help improve the interpretability and accuracy of prompt word quality assessment.

[0007] To achieve the above technical objectives, the technical solution adopted in this application includes: Firstly, this application provides a training method for a prompt word quality assessment model, comprising: acquiring prompt word training text, and performing text sequence segmentation on the prompt word training text to obtain a semantic accuracy sequence, an expression completeness sequence, and a logical coherence sequence of the prompt word training text; performing multi-dimensional parallel encoding fusion processing based on the semantic accuracy sequence, the expression completeness sequence, and the logical coherence sequence to obtain a target prompt word representation vector; performing dual-branch prediction fusion on the target prompt word representation vector to obtain fused prediction data; and updating the parameters of the initialized prompt word quality assessment model based on the fused prediction data to obtain a trained prompt word quality assessment model.

[0008] In addition, the method according to the above embodiments of this application may also have the following additional technical features: Further, in one embodiment of this application, the step of performing text sequence word segmentation on the prompt word training text to obtain the semantic accuracy sequence, expression integrity sequence, and logical coherence sequence of the prompt word training text includes: performing unified word segmentation on the prompt word training text to obtain a plurality of prompt words; performing semantic word extraction on all the prompt words to obtain the semantic accuracy sequence; performing information constraint word extraction on all the prompt words to obtain the expression integrity sequence; and performing structural word extraction on all the prompt words to obtain the logical coherence sequence.

[0009] Furthermore, in one embodiment of this application, the step of performing multi-dimensional parallel encoding fusion processing based on the semantic accuracy sequence, the expression completeness sequence, and the logical coherence sequence to obtain the target prompt word representation vector includes: obtaining a shared encoding network; inputting the semantic accuracy sequence, the expression completeness sequence, and the logical coherence sequence into the shared encoding network to obtain a first encoding embedding matrix corresponding to the semantic accuracy sequence, a second encoding embedding matrix corresponding to the expression completeness sequence, and a third encoding embedding matrix corresponding to the logical coherence sequence; and performing encoding fusion on the first encoding embedding matrix, the second encoding embedding matrix, and the third encoding embedding matrix to obtain the target prompt word representation vector.

[0010] Further, in one embodiment of this application, inputting the target sequence into the shared coding network to obtain a target coding embedding matrix includes: generating an embedding matrix from the target sequence to obtain an original embedding matrix; performing multi-head attention analysis on the original embedding matrix to obtain a weight matrix; performing attention weight fusion on the original embedding matrix based on the weight matrix to obtain an intermediate embedding matrix; and performing feature nonlinear transformation on the intermediate embedding matrix to obtain the target coding embedding matrix; wherein, if the target sequence is the semantic accuracy sequence, the target coding embedding matrix is ​​the first coding embedding matrix; or, if the target sequence is the expression integrity sequence, the target coding embedding matrix is ​​the second coding embedding matrix; or, if the target sequence is the logical coherence sequence, the target coding embedding matrix is ​​the third coding embedding matrix.

[0011] Furthermore, in one embodiment of this application, the step of encoding and fusing the first encoding embedding matrix, the second encoding embedding matrix, and the third encoding embedding matrix to obtain the target prompt word representation vector includes: performing vector adaptation processing based on the first encoding embedding matrix, the second encoding embedding matrix, and the third encoding embedding matrix to obtain a first embedding vector corresponding to the first encoding embedding matrix, a second embedding vector corresponding to the second encoding embedding matrix, and a third embedding vector corresponding to the third encoding embedding matrix; and performing attention-weighted fusion on the first embedding vector, the second embedding vector, and the third embedding vector to obtain the target prompt word representation vector.

[0012] Further, in one embodiment of this application, vector adaptation processing is performed based on the target encoding embedding matrix to obtain a target embedding vector corresponding to the target encoding embedding matrix, including: performing vector transformation on the target encoding embedding matrix to obtain an original embedding vector; performing vector feature optimization on the original embedding vector to obtain the target embedding vector; wherein, if the target encoding embedding matrix is ​​a first encoding embedding matrix, then the target embedding vector is the first embedding vector; or, if the target encoding embedding matrix is ​​a second encoding embedding matrix, then the target embedding vector is the second embedding vector; or, if the target encoding embedding matrix is ​​a third encoding embedding matrix, then the target embedding vector is the third embedding vector.

[0013] Furthermore, in this embodiment of the application, the step of performing dual-branch prediction fusion on the target prompt word representation vector to obtain fused prediction data includes: obtaining adaptive gating parameters; performing classification prediction on the target prompt word representation vector to obtain classification prediction data; performing ordinal prediction on the target prompt word representation vector to obtain ordinal prediction data; and performing gating fusion on the classification prediction data and the ordinal prediction data according to the adaptive gating parameters to obtain the fused prediction data.

[0014] Secondly, embodiments of this application provide a quality assessment method, including: obtaining target prompt word text to be quality assessed; inputting the target prompt word text into the above-mentioned trained prompt word quality assessment model for quality assessment, and obtaining the quality assessment result of the target prompt word text.

[0015] Thirdly, embodiments of this application provide a training system for a prompt word quality assessment model, comprising: a first processing unit, configured to acquire prompt word training text and perform text sequence segmentation on the prompt word training text to obtain a semantic accuracy sequence, an expression completeness sequence, and a logical coherence sequence of the prompt word training text; a second processing unit, configured to perform multi-dimensional parallel encoding fusion processing based on the semantic accuracy sequence, the expression completeness sequence, and the logical coherence sequence to obtain a target prompt word representation vector; a third processing unit, configured to perform dual-branch prediction fusion on the target prompt word representation vector to obtain fused prediction data; and a fourth processing unit, configured to update the parameters of an initialized prompt word quality assessment model based on the fused prediction data to obtain a trained prompt word quality assessment model.

[0016] Fourthly, embodiments of this application also provide an electronic device, including: at least one processor; at least one memory for storing at least one program; and when the at least one program is executed by the at least one processor, causing the at least one processor to implement the above-described method.

[0017] Fifthly, embodiments of this application also provide a computer-readable storage medium storing a processor-executable program, which, when executed by the processor, is used to implement the above-described method.

[0018] The advantages and beneficial effects of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application: The embodiment of this application discloses a training method and a quality assessment method for a prompt word quality assessment model. The training method acquires prompt word training text and performs text sequence segmentation on the prompt word training text to obtain a semantic accuracy sequence, an expression completeness sequence, and a logical coherence sequence. Based on the semantic accuracy sequence, the expression completeness sequence, and the logical coherence sequence, multi-dimensional parallel encoding and fusion processing is performed to obtain a target prompt word representation vector. The target prompt word representation vector is then subjected to dual-branch prediction fusion to obtain fused prediction data. Based on the fused prediction data, the parameters of the initialized prompt word quality assessment model are updated to obtain a trained prompt word quality assessment model. This training method, by performing multi-dimensional parallel encoding and fusion on factor sequences (such as semantic accuracy sequence, expression completeness sequence, or logical coherence sequence) of the prompt word training text, can achieve multi-dimensional analysis of key factors affecting prompt word quality, which is beneficial to improving the interpretability and accuracy of the model in prompt word quality assessment. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the following description is provided with accompanying drawings of the relevant technical solutions in the embodiments of this application or the prior art. It should be understood that the accompanying drawings described below are only for the purpose of clearly illustrating some embodiments of the technical solutions in this application. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0020] Figure 1 is a flowchart illustrating a training method for a prompt word quality assessment model provided in an embodiment of this application; Figure 2 is a framework diagram illustrating a training system for a prompt word quality assessment model provided in an embodiment of this application; Figure 3 is a structural diagram illustrating an electronic device provided in an embodiment of this application. Detailed Implementation

[0021] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application. The step numbers in the following embodiments are set only for ease of explanation, and there is no limitation on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0022] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0023] Unlike traditional documents, prompt words are essentially operational instructions for artificial intelligence models. They are typically short, highly semantically dense, and extremely sensitive to accuracy, completeness, and logical order. Defects in any of these dimensions (such as unclear semantic descriptions, missing constraints, or disordered structural logic) can directly impact the model's execution results. Therefore, evaluating the quality of prompt words requires not only an overall assessment of the text content but also the ability to decompose and analyze the key factors affecting quality.

[0024] Currently, related technologies typically assess the quality of prompt words based on pre-trained language models. However, since this approach often directly adopts common text classification paradigms, it fails to deeply adapt to the multidimensional and ordinal characteristics of prompt word quality. Consequently, it is difficult to analyze the multidimensional key factors affecting prompt word quality, and the interpretability and accuracy of prompt word quality assessment are unsatisfactory.

[0025] Furthermore, some related technologies rely on traditional neural networks (such as CNN, RNN, LSTM, etc.) to assess the quality of prompt words. While these methods can extract basic semantic features, they generally suffer from insufficient understanding of long texts and weak global context awareness. They also cannot adequately adapt to the multidimensional and ordinal characteristics of prompt word quality, resulting in unsatisfactory interpretability and accuracy of prompt word quality assessment.

[0026] It should be noted that the aforementioned related technologies are only used to assist in understanding the technical solutions of this application and do not mean that they belong to the publicly disclosed prior art.

[0027] In view of this, embodiments of this application provide a training method and a quality assessment method for a prompt word quality assessment model. The training method involves multi-dimensional parallel encoding and fusion of factor sequences (such as semantic accuracy sequences, expression completeness sequences, or logical coherence sequences) of different dimensions in the prompt word training text. Specifically, a shared encoding network is used to encode factor sequences of different dimensions in parallel. This allows semantic, completeness, and logical sequences to be represented in the same semantic space (and achieves collaborative modeling of semantic accuracy, expression completeness, and logical coherence), thereby ensuring that the features of the three dimensions are comparable, additive, and fusionable in space. This enables multi-dimensional analysis of key factors affecting prompt word quality, improving the interpretability and accuracy of the model in prompt word quality assessment.

[0028] Furthermore, since the quality levels of prompt words naturally exhibit ordinal relationships, with each level displaying both monotonicity and continuity, this method, in addition to traditional classification prediction, combines ordinal prediction with the prediction and analysis of the target prompt word representation vector. This allows for a more comprehensive consideration of the ordinal nature of prompt word quality, which is beneficial for improving the interpretability and accuracy of subsequent prompt word quality assessments by the model.

[0029] Referring to Figure 1, in this embodiment of the application, a training method for a prompt word quality assessment model includes: step 110, acquiring prompt word training text, and performing text sequence segmentation on the prompt word training text to obtain a semantic accuracy sequence, an expression completeness sequence, and a logical coherence sequence of the prompt word training text; in this embodiment of the application, the prompt word training text can be a publicly available prompt word dataset on the Internet, such as the publicly available argilla / 10k_prompts_ranked_mistral_large_responses prompt word dataset, the ProRate (Professional Rated Prompts) prompt word dataset, etc. The text sequence segmentation can be based on the three key factors affecting prompt word quality: semantic accuracy, expression completeness, and logical coherence, extracting several words from the prompt word training text, and constructing a semantic accuracy sequence, an expression completeness sequence, and a logical coherence sequence respectively. Semantic accuracy is used to evaluate whether the words (such as vocabulary and terminology) in the prompt words can accurately express the user's intent; completeness of expression is used to measure whether the prompt words provide sufficient information when completing a specific task (such as text generation task or classification task); and logical coherence is used to evaluate whether the organization and hierarchy of information within the prompt words are reasonable.

[0030] In some embodiments, the step of performing text sequence segmentation on the prompt word training text to obtain a semantic accuracy sequence, an expression completeness sequence, and a logical coherence sequence of the prompt word training text includes: performing unified segmentation on the prompt word training text to obtain a plurality of prompt words; extracting semantic words from all the prompt words to obtain the semantic accuracy sequence; extracting information constraint words from all the prompt words to obtain the expression completeness sequence; and extracting structural words from all the prompt words to obtain the logical coherence sequence.

[0031] In this embodiment, unified word segmentation can be performed on the prompt word training text. Various specific word segmentation operations exist, resulting in several prompt words. Next, several key terms related to semantic accuracy are selected from all prompt words, and a semantic accuracy sequence is constructed. Several information constraint words related to expressive integrity are selected from all prompt words, and an expressive integrity sequence is constructed. Finally, several structural words related to logical coherence are selected from all prompt words, and a logical coherence sequence is constructed. For example, if the prompt word training text is “Write a detailed report about the future development of artificial intelligence, including its major application areas, key technical challenges, and potential risks. The report should follow a structured order, starting with an overview, then presenting analysis, and finally giving conclusions.”, after unified word segmentation, key terms that directly reflect the core intent of the task can be identified, such as “artificial intelligence”, “application areas”, and “technical challenges”, thereby constructing a semantic accuracy sequence [“artificial intelligence”, “application areas”, “technical challenges”]. [challenges]; Identify informational words in the prompts that reflect content requirements or constraints, such as "future development", "potential risks", and "detailed report", to construct a sequence of complete expression ["future development", "potential risks", and "detailed report"]; and identify structural words that indicate the structure or sequence of the text, such as "starting with", "then", and "finally", to construct a sequence of logical coherence ["starting with", "then", and "finally"].

[0032] It should be noted that there are multiple ways to obtain the above three types of sequences. For example, it can be achieved through dictionary rules, manually constructed keyword sets, dependency parsing, saliency scoring models, or threshold filtering of attention weights of different dimensional features using pre-trained language models. This application will not elaborate further on these methods.

[0033] If the training text for the prompt word is “, then its semantic accuracy sequence is []; its expression completeness sequence is []; and its logical coherence sequence is [].

[0034] Step S120: Perform multi-dimensional parallel encoding fusion processing based on the semantic accuracy sequence, the expression completeness sequence, and the logical coherence sequence to obtain the target prompt word representation vector; In this embodiment of the application, multi-dimensional sequences can be encoded and fused to obtain a vector representation of the semantic accuracy sequence, the expression completeness sequence, and the logical coherence sequence in the same semantic space, denoted as the target prompt word representation vector.

[0035] In some embodiments, the step of performing multi-dimensional parallel encoding fusion processing based on the semantic accuracy sequence, the expression completeness sequence, and the logical coherence sequence to obtain the target prompt word representation vector includes: obtaining a shared encoding network; inputting the semantic accuracy sequence, the expression completeness sequence, and the logical coherence sequence into the shared encoding network to obtain a first encoding embedding matrix corresponding to the semantic accuracy sequence, a second encoding embedding matrix corresponding to the expression completeness sequence, and a third encoding embedding matrix corresponding to the logical coherence sequence; further, inputting the target sequence into the shared encoding network to obtain a target encoding embedding matrix includes: generating an embedding matrix for the target sequence to obtain an original embedding matrix; performing multi-head attention analysis on the original embedding matrix to obtain a weight matrix; performing attention weight fusion on the original embedding matrix based on the weight matrix to obtain an intermediate embedding matrix; and performing feature nonlinear transformation on the intermediate embedding matrix to obtain the target encoding embedding matrix; in this embodiment, the shared encoder can be a BERT backbone network based on a parameter sharing mechanism, with the three types of sequences input in parallel to a shared encoder with the same structural parameters to obtain an encoding embedding matrix corresponding to each sequence. Specifically, if the target sequence is a semantically accurate sequence, the embedding matrix can be generated by converting the text of the target sequence into word vector form through a pre-trained BERT word segmenter, and then mapping the target sequence in word vector form to the hidden layer dimension of the model through a linear transformation to form an embedding matrix of [sequence length, hidden layer dimension], denoted as the original embedding matrix.

[0036] Understandably, multi-head attention analysis can involve inputting the original embedding matrix into a multi-head attention layer to obtain the weight matrix output by the multi-head attention layer. Then, attention weight fusion can involve performing a weighted summation operation between the weight matrix and the original embedding matrix and normalizing it to enhance the extraction of key information. The normalized original embedding matrix is ​​then denoted as the intermediate embedding matrix.

[0037] It should be noted that, in the first embodiment, the feature nonlinear transformation can be performed by inputting the intermediate embedding matrix into the feedforward neural network (FFN) and determining the output of the feedforward neural network as the first encoding embedding matrix; or, in the second embodiment, the feature nonlinear transformation can be performed by inputting the intermediate embedding matrix into the feedforward neural network (FFN) and performing the feature nonlinear transformation, and then normalizing the output of the feedforward neural network to alleviate gradient vanishing, and then determining the normalized output of the feedforward neural network as the first encoding embedding matrix.

[0038] It is worth mentioning that the target sequence is the content of the sequence of completeness or logical coherence, which is similar to the aforementioned target sequence being the content of the sequence of semantic accuracy, and can be easily deduced by analogy.

[0039] The first encoding embedding matrix, the second encoding embedding matrix, and the third encoding embedding matrix are encoded and fused to obtain the target prompt word representation vector.

[0040] Further, the step of encoding and fusing the first encoding embedding matrix, the second encoding embedding matrix, and the third encoding embedding matrix to obtain the target prompt word representation vector includes: performing vector adaptation processing based on the first encoding embedding matrix, the second encoding embedding matrix, and the third encoding embedding matrix to obtain a first embedding vector corresponding to the first encoding embedding matrix, a second embedding vector corresponding to the second encoding embedding matrix, and a third embedding vector corresponding to the third encoding embedding matrix; further, performing vector adaptation processing based on the target encoding embedding matrix to obtain a target embedding vector corresponding to the target encoding embedding matrix includes: performing vector transformation on the target encoding embedding matrix to obtain an original embedding vector; and performing vector feature optimization on the original embedding vector to obtain the target embedding vector; in this embodiment, the vector adaptation processing may be performed on the first encoding embedding matrix, the second encoding embedding matrix, and the third encoding embedding matrix respectively to refine the features of each dimension, thereby improving the accuracy of the model in evaluating the quality of prompt words.

[0041] Specifically, if the target encoding embedding matrix is ​​the first encoding embedding matrix, vector transformation can be performed on the first encoding embedding matrix using masked mean pooling to obtain a fixed-length feature vector, denoted as the original embedding vector. Vector feature optimization can be performed by inputting the original embedding vector into an optimization module, which sequentially includes a first fully connected layer, a GeLU activation function layer, and a second fully connected layer. The first fully connected layer is used to reduce the dimensionality of the input original embedding vector to compress the hidden layer dimension of the original embedding vector. The GeLU activation function layer introduces a nonlinear transformation to enhance the nonlinear expressive power of the feature information contained in the embedding vector. The second fully connected layer is used to restore the original embedding vector to its original hidden layer dimension and output the first embedding vector.

[0042] It is understandable that the target encoding embedding matrix is ​​the content of the second encoding embedding matrix or the third encoding embedding matrix, which is similar to the aforementioned target encoding embedding matrix being the content of the first encoding embedding matrix, and can be easily deduced by analogy.

[0043] Attention-weighted fusion is performed on the first embedding vector, the second embedding vector, and the third embedding vector to obtain the target prompt word representation vector.

[0044] In this embodiment, attention-weighted fusion can be based on an attention mechanism, which involves weighting the first embedding vector, the second embedding vector, and the third embedding vector, and then determining the resulting vector representation as the target prompt word representation vector. There are various implementations of weighting operations based on the attention mechanism, which will not be elaborated here.

[0045] Step S130: Perform dual-branch prediction fusion on the target prompt word representation vector to obtain fused prediction data; In this embodiment of the application, the quality score of the target prompt word representation vector can be predicted by a dual-branch prediction head to obtain fused prediction data. The dual-branch prediction head includes a classification prediction head and an ordinal prediction head.

[0046] In some embodiments, the step of performing bi-branch prediction fusion on the target prompt word representation vector to obtain fused prediction data includes: obtaining adaptive gating parameters; performing classification prediction on the target prompt word representation vector to obtain classification prediction data; performing ordinal prediction on the target prompt word representation vector to obtain ordinal prediction data; and performing gating fusion on the classification prediction data and the ordinal prediction data according to the adaptive gating parameters to obtain the fused prediction data.

[0047] In this embodiment, adaptive gating parameters are used to contribute to classification and regression. Classification prediction can be achieved by inputting the target prompt word representation vector into the classification prediction head in the two-branch prediction head, and outputting the discrete probability distribution of the classification through the classification prediction head, denoted as classification prediction data; while ordinal prediction can be achieved by inputting the target prompt word representation vector into the ordinal prediction head in the two-branch prediction head, mapping the target prompt word feature vector to latent scores through the ordinal prediction head, and converting the latent scores into ordinal probabilities through a logistic function, denoted as ordinal prediction data.

[0048] It is understood that gated fusion can be achieved by adaptively gating parameters to fuse classification prediction data and ordinal prediction data into a final fused prediction probability, denoted as fused prediction data. For example, the classification prediction data, ordinal prediction data, and fused prediction data in the embodiments of this application can be represented as:

[0049] in, For classification and prediction data; For the Softmax function; and These are the learnable parameters of the classification prediction head, which can be obtained through parameter updates during model training. Vector representation of target prompt words; Potential scores; and Learnable parameters for the classification prediction head; This is the first representation of ordinal prediction data; For the Sigmoid function; As a variable; and These are the k-th and (k-1)-th learnable thresholds, respectively, which can also be obtained during parameter updates during model training. For adaptive gating parameters; The learnable weight matrix used to generate adaptive gating parameters incorporates the target cue word representation vector during the gating fusion process. The mapping is a gating signal that distinguishes the contribution ratio of classification prediction and ordinal prediction; To integrate the predicted data.

[0050] It should be noted that in practical applications, a quality level can be assigned to each probability range of the fused prediction data. This quality level can be "Excellent," "Good," "Pass," or a cue word quality score (e.g., 1-5). Specifically, in the first embodiment, a specific value of the fused prediction data greater than or equal to 0.9 can be considered an "Excellent" quality level; or, in the second embodiment, a specific value of the fused prediction data greater than or equal to 0.85 can be considered the highest cue word quality score. For example, when the cue word quality score is 1 to 5, the quality level corresponding to the fused prediction data is "5."

[0051] Step S140: Update the parameters of the initialized prompt word quality assessment model based on the fused prediction data to obtain the trained prompt word quality assessment model.

[0052] In this embodiment, the accuracy of the model's predictions can be evaluated based on the real labels of the fused prediction data and the training text of the prompt words, thereby updating the model's parameters. Specifically, a loss function combined with the real labels can be used to measure the loss value of the fused prediction data, denoted as the target loss value. The model's parameters are then updated using the backpropagation algorithm based on the target loss value. After several iterations, a well-trained prompt word quality assessment model can be obtained. The specific number of iterations can be preset, or training can be considered complete when the accuracy requirement on the test set is met.

[0053] It is understandable that there are many types of commonly used loss functions, such as 0-1 loss function, squared loss function, absolute loss function, log loss function, cross-entropy loss function, EMD loss function, Pairwise Ranking loss function, and R-Drop loss function. In practical applications, any one of these loss functions can be used alone or in combination. For example, this application uses EMD loss function, Pairwise Ranking loss function, and R-Drop loss function as examples. The EMD loss function measures the cumulative distribution difference between the fused predicted data and the true labels; the Pairwise Ranking loss function measures the consistency of score ranking between training samples of different quality levels; and the R-Drop loss function constrains the consistency of the predicted distribution after two random dropouts of the same prompt word using KL divergence. The target loss value can be expressed as:

[0054] in, The target loss value; This is the symbolic representation of the R-Drop loss function; This is the symbolic representation of the Pairwise Ranking loss function; The symbolic representation of the EMD loss function; , and These are weighting parameters, verified by experimental data. , and The optimal weighting values ​​are; The total number of quality grade categories. For category index (1 to K). Auxiliary indices (1 to k) for internal cumulative summation; and These represent two predicted probability distributions obtained after two random dropouts of the same input sample from the training text containing the prompt words. This is a functional representation of the Kullback–Leibler divergence; and These are the latent scores of the i-th and j-th input samples in the prompt word training text, respectively, output by the ordinal prediction head; This is the activation function for Softlus; This is a temperature coefficient used to control the smoothness of the Softplus function; its specific value can be set according to the actual situation. Used to represent samples The true quality level is greater than the sample The true quality level; This refers to the batch size. The index of the b-th input sample of the training text for the prompt word; Indicates the first The input sample at the th th The fusion prediction probability at each quality level, i.e., the fusion prediction data; This indicates that the target distribution obtained after smoothing the true quality level of the sample is at the th order. Values ​​at each quality level.

[0055] It should be noted that, due to the inherent subjectivity and significant score fluctuations in prompt word quality labeling, this embodiment combines the EMD loss function, the Pairwise Ranking loss function, and the R-Drop loss function. These functions jointly constrain the model at three levels: distribution, order, and stability. Specifically, the EMD loss function narrows the gap between the predicted and true distributions; the Pairwise Ranking loss function enhances the relative quality order among different prompt words; and the R-Drop loss function significantly improves the model's stability during the inference phase, reducing score fluctuations caused by random perturbations. This composite loss structure effectively addresses prompt word labeling noise, improving the accuracy and interpretability of the model's prompt word quality assessment.

[0056] The following describes the prompt word quality assessment model mentioned in the embodiments of this application based on experimental data: Using RNN, CNN, LSTM, and BERT-FC (traditional BERT + fully connected output) as benchmark models, comparative experiments were conducted on the publicly available ProRate prompt word dataset (containing 5000 labeled prompt words, covering text generation and classification tasks). The experimental environment was PyTorch 2.6 and an NVIDIA RTX 3080 GPU. Training parameters were set to a learning rate of 5e-5, a batch size of 16, and 3 training epochs. Evaluation metrics covered three dimensions: semantic accuracy, expressive completeness, and logical coherence. The experimental results are shown in Tables 1 to 3 below: Table 1 - Performance Comparison of Each Model in Semantic Accuracy Dimension

[0057] As shown in Table 1, the prompt word quality assessment model mentioned in the embodiments of this application outperforms the benchmark model in all indicators of semantic accuracy. Specifically, Acc is improved by 6.3% compared to BERT-FC, F1-W is improved by 6.2% compared to BERT-FC, and Prec-M is improved by 12.2% compared to BERT-FC. This proves that the prompt word quality assessment model mentioned in the embodiments of this application can accurately capture the semantic details of prompt words and effectively reduce the risk of misunderstanding of intent.

[0058] Table 2 - Performance Comparison of Each Model in Expression Completeness Dimension

[0059] As shown in Table 2, the prompt word quality assessment model mentioned in the embodiments of this application has the highest QWK in the expression completeness dimension, reaching 0.476, and the lowest RMSE and MAE, at 0.687 and 0.624, respectively. The RMSE is 7.2% lower than BERT-FC, which proves that the prompt word quality assessment model mentioned in the embodiments of this application can effectively measure the sufficiency of prompt word information and reduce AI output bias caused by missing information.

[0060] Table 3 - Performance Comparison of Each Model in Logical Coherence Dimension

[0061] As shown in Table 3, the prompt quality assessment model mentioned in the embodiments of this application has the highest Spearman and Kendall coefficients in the logical coherence dimension, reaching 0.554 and 0.439 respectively, and the ECE is only 0.039, which is 75.3% lower than BERT-FC. This proves that the prompt quality assessment model mentioned in the embodiments of this application has significantly improved the ranking consistency and confidence of the assessment results, and has stronger interpretability.

[0062] In this embodiment of the application, a quality assessment method includes: obtaining target prompt text to be quality assessed; inputting the target prompt text into the above-mentioned trained prompt text quality assessment model for quality assessment, and obtaining the quality assessment result of the target prompt text.

[0063] In this embodiment, the target prompt text to be evaluated can be input into the trained prompt quality evaluation model. The trained prompt quality evaluation model is used to perform quality evaluation to obtain the quality evaluation result of the target prompt text. In practical applications, this quality evaluation result can help users optimize the prompts input into AI models (such as large language models LLM), thereby improving the task output quality of AI models.

[0064] Referring to Figure 2, a training system for a prompt word quality assessment model proposed in this embodiment includes a first processing unit 101, used to acquire prompt word training text and perform text sequence segmentation on the prompt word training text to obtain a semantic accuracy sequence, an expression completeness sequence, and a logical coherence sequence of the prompt word training text; a second processing unit 102, used to perform multi-dimensional parallel encoding fusion processing based on the semantic accuracy sequence, the expression completeness sequence, and the logical coherence sequence to obtain a target prompt word representation vector; a third processing unit 103, used to perform dual-branch prediction fusion on the target prompt word representation vector to obtain fused prediction data; and a fourth processing unit 104, used to update the parameters of the initialized prompt word quality assessment model based on the fused prediction data to obtain a trained prompt word quality assessment model.

[0065] Referring to FIG3, an embodiment of this application also provides an electronic device, including: at least one processor 201; at least one memory 202 for storing at least one program; when the at least one program is executed by the at least one processor 201, the at least one processor 201 implements the above-described method embodiment.

[0066] Similarly, it can be understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0067] This application also provides a computer-readable storage medium storing a program executable by a processor 201, which, when executed by the processor 201, is used to implement the above-described method embodiments.

[0068] Similarly, the content of the above method embodiments is applicable to the present computer-readable storage medium embodiments. The specific functions implemented by the present computer-readable storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0069] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0070] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods.

[0071] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this application are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and sub-operations described as part of a larger operation are executed independently.

[0072] Furthermore, although this application is described in the context of functional modules, it should be understood that, unless otherwise stated to the contrary, one or more of the functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding this application. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional technology for an engineer. Therefore, those skilled in the art can implement the application set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of this application, which is determined by the full scope of the appended claims and their equivalents.

[0073] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods in the embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0074] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0075] More specific examples (a non-exhaustive list) of computer-readable media include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0076] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0077] In the foregoing description of this specification, the references to terms such as "one embodiment," "another embodiment," or "some embodiments," etc., indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0078] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

[0079] The above is a detailed description of the preferred embodiments of this application, but this application is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A training method for a prompt word quality assessment model, characterized in that, include: The process involves acquiring training text for prompt words and performing text sequence segmentation on the training text to obtain semantic accuracy sequence, expression completeness sequence, and logical coherence sequence. Based on these sequences, multi-dimensional parallel encoding fusion processing is performed to obtain target prompt word representation vectors. These target prompt word representation vectors are then subjected to dual-branch prediction fusion to obtain fused prediction data. Finally, the parameters of the initialized prompt word quality assessment model are updated based on the fused prediction data to obtain a trained prompt word quality assessment model.

2. The method according to claim 1, characterized in that, The step of performing text sequence segmentation on the prompt word training text to obtain the semantic accuracy sequence, expression completeness sequence, and logical coherence sequence of the prompt word training text includes: performing unified segmentation on the prompt word training text to obtain several prompt words; extracting semantic words from all the prompt words to obtain the semantic accuracy sequence; extracting information constraint words from all the prompt words to obtain the expression completeness sequence; and extracting structural words from all the prompt words to obtain the logical coherence sequence.

3. The method according to claim 1, characterized in that, The step of performing multi-dimensional parallel encoding fusion processing based on the semantic accuracy sequence, the expression completeness sequence, and the logical coherence sequence to obtain the target prompt word representation vector includes: obtaining a shared encoding network; inputting the semantic accuracy sequence, the expression completeness sequence, and the logical coherence sequence into the shared encoding network to obtain a first encoding embedding matrix corresponding to the semantic accuracy sequence, a second encoding embedding matrix corresponding to the expression completeness sequence, and a third encoding embedding matrix corresponding to the logical coherence sequence; and performing encoding fusion on the first encoding embedding matrix, the second encoding embedding matrix, and the third encoding embedding matrix to obtain the target prompt word representation vector.

4. The method according to claim 3, characterized in that, The target sequence is input into the shared coding network to obtain a target coding embedding matrix, including: generating an embedding matrix from the target sequence to obtain an original embedding matrix; performing multi-head attention analysis on the original embedding matrix to obtain a weight matrix; performing attention weight fusion on the original embedding matrix based on the weight matrix to obtain an intermediate embedding matrix; and performing feature nonlinear transformation on the intermediate embedding matrix to obtain the target coding embedding matrix; wherein, if the target sequence is the semantic accuracy sequence, the target coding embedding matrix is ​​the first coding embedding matrix; or, if the target sequence is the expression integrity sequence, the target coding embedding matrix is ​​the second coding embedding matrix; or, if the target sequence is the logical coherence sequence, the target coding embedding matrix is ​​the third coding embedding matrix.

5. The method according to claim 3, characterized in that, The step of encoding and fusing the first encoding embedding matrix, the second encoding embedding matrix, and the third encoding embedding matrix to obtain the target prompt word representation vector includes: performing vector adaptation processing based on the first encoding embedding matrix, the second encoding embedding matrix, and the third encoding embedding matrix to obtain a first embedding vector corresponding to the first encoding embedding matrix, a second embedding vector corresponding to the second encoding embedding matrix, and a third embedding vector corresponding to the third encoding embedding matrix; and performing attention-weighted fusion on the first embedding vector, the second embedding vector, and the third embedding vector to obtain the target prompt word representation vector.

6. The method according to claim 5, characterized in that, Based on the target encoding embedding matrix, vector adaptation processing is performed to obtain a target embedding vector corresponding to the target encoding embedding matrix. This includes: performing vector transformation on the target encoding embedding matrix to obtain an original embedding vector; and performing vector feature optimization on the original embedding vector to obtain the target embedding vector. Wherein, if the target encoding embedding matrix is ​​a first encoding embedding matrix, then the target embedding vector is the first embedding vector; or, if the target encoding embedding matrix is ​​a second encoding embedding matrix, then the target embedding vector is the second embedding vector; or, if the target encoding embedding matrix is ​​a third encoding embedding matrix, then the target embedding vector is the third embedding vector.

7. The method according to any one of claims 1-6, wherein performing bi-branch prediction fusion on the target prompt word representation vector to obtain fused prediction data includes: Obtain adaptive gating parameters; The target prompt word representation vector is classified and predicted to obtain classification prediction data; The target prompt word representation vector is subjected to ordinal prediction to obtain ordinal prediction data; the classification prediction data and the ordinal prediction data are subjected to gating fusion according to the adaptive gating parameters to obtain the fused prediction data.

8. A quality assessment method, characterized in that, include: Obtain the target cue text to be evaluated for quality; The target prompt text is input into the trained prompt quality assessment model as described in any one of claims 1-7 for quality assessment, and the quality assessment result of the target prompt text is obtained.

9. A training system for a prompt word quality assessment model, characterized in that, include: The first processing unit is used to acquire the prompt word training text and perform text sequence segmentation on the prompt word training text to obtain the semantic accuracy sequence, expression integrity sequence and logical coherence sequence of the prompt word training text; The second processing unit is used to perform multi-dimensional parallel encoding fusion processing based on the semantic accuracy sequence, the expression integrity sequence, and the logical coherence sequence to obtain the target prompt word representation vector; The third processing unit is used to perform dual-branch prediction fusion on the target prompt word representation vector to obtain fused prediction data; the fourth processing unit is used to update the parameters of the initialized prompt word quality assessment model according to the fused prediction data to obtain the trained prompt word quality assessment model.

10. An electronic device, characterized in that, include: At least one processor; At least one memory for storing at least one program; when said at least one program is executed by said at least one processor, such that said at least one processor implements the method as claimed in any one of claims 1-8.

Images