A security assessment method and device for generative artificial intelligence

Abstract

The present invention discloses a security assessment method and device for generative artificial intelligence, including: constructing a training sample data set, which includes a plurality of training sample subsets in different fields; predicting and processing the training sample subsets in different fields through a preset prediction model to obtain the rejection rate, sentence fluency, description accuracy and content safety corresponding to the training sample subsets in different fields respectively; assigning corresponding weights to the rejection rate, sentence fluency, description accuracy and content safety based on the security assessment requirements to calculate the rejection rate score, sentence fluency score, description accuracy score and content safety score; determining the security assessment result based on the rejection rate score, sentence fluency score, description accuracy score and content safety score. Multi-dimensional security assessment analysis can be realized, and the model can ensure the timeliness and effectiveness of the assessment results through continuous learning and self-optimization, reducing the time and cost of manual participation.

Description

Technical Field

[0001] The present invention relates to the field of network security technology, and in particular to a security assessment method and device for generative artificial intelligence. Background Art

[0002] Generative AI refers to the technology that generates text, pictures, sounds, videos, codes, etc. based on algorithms, models, and rules. It changes the way humans interact with technology and promotes innovation in the development of artificial intelligence.

[0003] However, current generative AI still has limitations in processing complex contexts and understanding user intent. This may result in the model being unable to accurately understand user needs, thereby generating content that does not meet user expectations. During the training and application of the model, sensitive information and privacy of users may be involved, raising issues of data security and privacy protection. In particular, the lack of expertise in certain specific areas leads to poor performance on issues in specific industries or fields. For example, there may be limitations in the understanding of terminology and normative language in the field of cybersecurity and its sub-fields.

[0004] Currently, statistical-based security assessment methods are generally used, which usually require a large amount of labeled data for model training. However, in practical applications, labeled data is often difficult to obtain, and the labeling process is time-consuming and labor-intensive. At the same time, the cost of model training is relatively high, requiring powerful computing resources and time investment. More importantly, even after the model training is completed, its ability to identify new security threats remains to be verified. Summary of the invention

[0005] Based on this, it is necessary to provide a security assessment method, device, computer equipment and storage medium for generative artificial intelligence to address the above technical issues, so as to solve at least one problem existing in the above-mentioned prior art.

[0006] In a first aspect, the embodiment of the present application is implemented as follows: a security assessment method for generative artificial intelligence is provided, comprising:

[0007] Constructing a training sample data set, wherein the training sample data set includes a plurality of training sample subsets in different fields;

[0008] Predicting the subsets of training samples in different fields by using a preset prediction model to obtain the rejection rate, sentence fluency, description accuracy and content safety corresponding to the subsets of training samples in different fields respectively;

[0009] Based on the security assessment requirements, corresponding weights are assigned to the rejection rate, sentence fluency, description accuracy, and content safety, and based on the weights, a rejection rate score, a sentence fluency score, a description accuracy score, and a content safety score are calculated;

[0010] A security assessment result is determined based on the rejection rate score, sentence fluency score, description accuracy score, and content safety score.

[0011] In one embodiment, the predictive processing of the different domain training sample subsets by using a preset prediction model to obtain the rejection rate, sentence fluency, description accuracy and content safety corresponding to the different domain training sample subsets respectively includes:

[0012] Predicting the training sample subsets in different fields by using a preset rejection rate prediction model to obtain the rejection rates corresponding to the training sample subsets in different fields;

[0013] Predicting the subsets of training samples in different fields by using a preset sentence fluency evaluation model to obtain the sentence fluency corresponding to the subsets of training samples in different fields;

[0014] Predicting the subsets of training samples in different fields by using a preset accuracy prediction model to obtain the description accuracy corresponding to the subsets of training samples in different fields;

[0015] The different domain training sample subsets are predicted by a preset content security prediction model to obtain the content security corresponding to the different domain training sample subsets.

[0016] In one embodiment, predicting the training sample subsets in different fields by using a preset rejection rate prediction model to obtain the rejection rates corresponding to the training sample subsets in different fields includes:

[0017] Preprocessing the training sample data in each of the training sample subsets to obtain a preprocessed training sample subset;

[0018] Based on the preprocessed training sample subset, the preset refusal rate prediction model is trained to obtain a refusal prediction result corresponding to each training sample data;

[0019] Based on the refusal to answer prediction result, the refusal rates corresponding to the training sample subsets in different fields are obtained.

[0020] In one embodiment, the preset rejection rate prediction model includes a plurality of decision trees, and the training of the preset rejection rate prediction model based on the preprocessed training sample subset includes:

[0021] Randomly selecting training sample data from the preprocessed training sample subset;

[0022] Inputting randomly selected training sample data into the decision tree to obtain the classification results output by each decision tree;

[0023] The classification result with the largest number of occurrences is selected from the classification results as the final classification result of the randomly selected training sample data.

[0024] In one embodiment, the predicting of the different domain training sample subsets by using a preset sentence fluency evaluation model to obtain the sentence fluency corresponding to the different domain training sample subsets includes:

[0025] Preprocessing the training sample data in each of the training sample subsets to obtain a preprocessed training sample subset;

[0026] Based on the training sample subset, the preset sentence fluency model is trained to obtain a fluency prediction result corresponding to each training sample data;

[0027] Based on the fluency prediction result, the sentence fluency corresponding to the training sample subsets in different fields is obtained.

[0028] In one embodiment, the predicting of the different domain training sample subsets by a preset accuracy prediction model to obtain the description accuracy corresponding to the different domain training sample subsets includes:

[0029] Preprocessing the training sample data in each of the training sample subsets to obtain a preprocessed training sample subset;

[0030] Performing a linear transformation on the training sample subset to obtain a query vector, a key vector, and a value vector;

[0031] Perform multi-head attention calculation based on the query vector, the key vector, and the value vector to obtain multiple prediction results;

[0032] The plurality of prediction results are concatenated, and the concatenated prediction results are linearly transformed to obtain a multi-head attention prediction result.

[0033] Based on the multi-head attention prediction results, the description accuracy corresponding to the training sample subsets in different fields is obtained.

[0034] In one embodiment, the predicting the different domain training sample subsets by using a preset content security prediction model to obtain the content security corresponding to the different domain training sample subsets includes:

[0035] Preprocessing the training sample data in each of the training sample subsets to obtain a preprocessed training sample subset;

[0036] Based on the linear regression model, linear regression processing is performed on the training sample subset to obtain an output result;

[0037] The output result is calculated by a preset activation function to obtain the content security corresponding to each training sample data;

[0038] Based on the content security corresponding to each training sample data, the content security corresponding to the training sample subsets in different fields is obtained.

[0039] In one embodiment, preprocessing the training sample data in each of the training sample subsets to obtain the preprocessed training sample subsets includes:

[0040] Based on the requirements of each model, the training sample data in each of the training sample subsets is subjected to corresponding standardization processing;

[0041] Perform feature extraction on the standardized training sample data to obtain a feature subset;

[0042] Based on the feature subset, correspondingly labeling the training sample data;

[0043] Based on the labeled training sample data, the preprocessed training sample subset is generated.

[0044] In one embodiment, the security assessment result is obtained based on the rejection rate score, the sentence fluency score, the description accuracy score, and the content safety score, including:

[0045] Calculate a safety assessment score based on the refusal rate score, sentence fluency score, description accuracy score, and content safety score;

[0046] Based on the safety assessment score, the safety assessment result is determined.

[0047] In a second aspect, a security assessment device for generative artificial intelligence is provided, the device comprising:

[0048] A training sample data construction unit, used to construct a training sample data set, wherein the training sample data set includes a plurality of training sample subsets in different fields;

[0049] A prediction unit, configured to perform prediction processing on the training sample subsets in different fields by using a preset prediction model, so as to obtain the rejection rate, sentence fluency, description accuracy and content safety corresponding to the training sample subsets in different fields respectively;

[0050] A weight allocation unit, configured to allocate corresponding weights to the rejection rate, sentence fluency, description accuracy, and content safety based on security assessment requirements, and calculate a rejection rate score, a sentence fluency score, a description accuracy score, and a content safety score based on the weights;

[0051] The security assessment result determination unit is used to determine the security assessment result based on the rejection rate score, the sentence fluency score, the description accuracy score and the content safety score.

[0052] In a third aspect, a computer device is provided, comprising a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein the processor implements the above-mentioned generative artificial intelligence security assessment method when executing the computer-readable instructions.

[0053] In a fourth aspect, a readable storage medium is provided, wherein when the computer-readable instructions are executed by one or more processors, the one or more processors execute the above-mentioned generative artificial intelligence security assessment method.

[0054] The above-mentioned security assessment method, device, computer equipment and storage medium of generative artificial intelligence, the method of which is implemented, includes: constructing a training sample data set, the training sample data set includes a plurality of training sample subsets in different fields; predicting and processing the training sample subsets in different fields through a preset prediction model to obtain the rejection rate, sentence fluency, description accuracy and content safety corresponding to the training sample subsets in different fields respectively; assigning corresponding weights to the rejection rate, sentence fluency, description accuracy and content safety based on the security assessment requirements, and calculating the rejection rate score, sentence fluency score, description accuracy score and content safety score based on the weights; determining the security assessment result based on the rejection rate score, sentence fluency score, description accuracy score and content safety score. In the embodiment of the present application, a multi-dimensional security assessment analysis is performed from the rejection rate, sentence fluency, description accuracy and content safety by combining machine learning, deep learning and other technologies, and the model has adaptive learning ability, and can continuously update the assessment model through continuous learning and self-optimization to cope with new challenges, ensure the timeliness and effectiveness of the assessment results, and reduce the time and cost of manual participation. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0056] Figure 1 It is a schematic diagram of a process of a security assessment method of generative artificial intelligence in one embodiment of the present invention;

[0057] Figure 2In one embodiment of the present invention, a schematic diagram of using a recurrent connection to draw a recurrent neural network and its loss calculation;

[0058] Figure 3 is a schematic diagram of a structure of a security assessment device for generative artificial intelligence in one embodiment of the present invention;

[0059] Figure 4 is a schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0060] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] In one embodiment, if Figure 1 As shown, a security assessment method for generative artificial intelligence is provided, comprising the following steps:

[0062] In step S110, a training sample data set is constructed, wherein the training sample data set includes a plurality of training sample subsets in different fields;

[0063] In the embodiment of the present application, data from multiple different fields can be collected, which can specifically include multi-source data in the fields of network security, history, humanities, etc. Exemplarily, connections can be established with multiple different platforms, such as a network security platform, to obtain network security-related data, such as professional terms, network abnormality data, network normal data, etc. The data collected for each field is placed in a data set as a subset of training sample data for the corresponding field.

[0064] It should be noted that each training sample data subset may include a large amount of training sample data, and the training sample data is text data.

[0065] In step S120, the different domain training sample subsets are predicted by a preset prediction model to obtain the rejection rate, sentence fluency, description accuracy and content safety corresponding to the different domain training sample subsets respectively;

[0066] In an embodiment of the present application, the preset prediction model may include multiple ones, such as a rejection rate prediction model, a fluency prediction model, an accuracy prediction model, and a content security prediction model, and the training sample data is respectively input into the corresponding models to predict the rejection rate, sentence fluency, description accuracy, and content security corresponding to the training sample subsets in different fields.

[0067] Understandably, the rejection rate can indicate the rate at which content is refused to be answered or processed. A high rejection rate may indicate that there are problems with the content, such as sensitivity, controversy, or inappropriateness. Sentence fluency can be used to measure the fluency of sentence grammar and expression. The smoother the sentence, the more understandable and acceptable the text content is, thereby improving its security assessment. Description accuracy can reflect the consistency between the content description and the actual situation. High accuracy means that the information is clearly conveyed, which reduces misunderstandings and ambiguities and enhances the security of the content. Content security can directly evaluate the compliance and security of the content. High security means that the content is healthier, more compliant, and has lower risks.

[0068] In step S130, based on the security assessment requirements, corresponding weights are assigned to the rejection rate, sentence fluency, description accuracy, and content safety, and based on the weights, a rejection rate score, a sentence fluency score, a description accuracy score, and a content safety score are calculated;

[0069] In an embodiment of the present application, the security assessment requirements may specifically include the business scenario of the model for determination. For example, if it is a business scenario of laws and regulations, the content security and description accuracy may be assigned a larger weight. If it is a business scenario of composition or article, the sentence fluency may be assigned a larger weight.

[0070] For example, are the weight coefficients of rejection rate, sentence fluency, description accuracy and content security, respectively, and R, F, A, and C represent the rejection rate, sentence fluency, description accuracy, and content safety, respectively. The rejection rate score, sentence fluency score, description accuracy score, and content safety score are , , as well as .

[0071] In step S140, a security assessment result is determined based on the rejection rate score, the sentence fluency score, the description accuracy score, and the content safety score.

[0072] In an embodiment of the present application, after obtaining the rejection rate score, sentence fluency score, description accuracy score and content safety score, the security assessment index can be calculated through a pre-built security assessment index algorithm model, and then when the security assessment index is greater than a preset threshold, the model can be considered to meet the security assessment requirements, otherwise the weight needs to be adjusted in order to optimize the model.

[0073] The embodiment of the present application provides a security assessment method for generative artificial intelligence, including: constructing a training sample data set, the training sample data set includes a plurality of training sample subsets in different fields; predicting and processing the training sample subsets in different fields through a preset prediction model to obtain the rejection rate, sentence fluency, description accuracy and content safety corresponding to the training sample subsets in different fields respectively; assigning corresponding weights to the rejection rate, sentence fluency, description accuracy and content safety based on the security assessment requirements, and calculating the rejection rate score, sentence fluency score, description accuracy score and content safety score based on the weights; determining the security assessment result based on the rejection rate score, sentence fluency score, description accuracy score and content safety score. In the embodiment of the present application, a multi-dimensional security assessment analysis is performed from the rejection rate, sentence fluency, description accuracy and content safety by combining machine learning, deep learning and other technologies, and the model has adaptive learning ability, and can continuously update the assessment model through continuous learning and self-optimization to cope with new challenges, ensure the timeliness and effectiveness of the assessment results, and reduce the time and cost of manual participation.

[0074] In one embodiment of the present application, the prediction processing of the different domain training sample subsets by a preset prediction model to obtain the rejection rate, sentence fluency, description accuracy and content safety corresponding to the different domain training sample subsets respectively includes:

[0075] Predicting the training sample subsets in different fields by using a preset rejection rate prediction model to obtain the rejection rates corresponding to the training sample subsets in different fields;

[0076] Predicting the subsets of training samples in different fields by using a preset sentence fluency evaluation model to obtain the sentence fluency corresponding to the subsets of training samples in different fields;

[0077] Predicting the subsets of training samples in different fields by using a preset accuracy prediction model to obtain the description accuracy corresponding to the subsets of training samples in different fields;

[0078] The different domain training sample subsets are predicted by a preset content security prediction model to obtain the content security corresponding to the different domain training sample subsets.

[0079] Optionally, the rejection rate prediction model can be a random forest algorithm model, the fluency prediction model can be a recurrent neural network model, such as RNN, the accuracy prediction model can be a multi-head attention mechanism algorithm model in deep learning, and the content security prediction model can be a logistic regression algorithm model. The security analysis method based on the random forest algorithm model, recurrent neural network model, multi-head attention mechanism model, and logistic regression model realizes an automated deep security assessment system based on the four key elements of rejection rate, sentence fluency, description accuracy, and content security. The quality and accuracy of generative artificial intelligence security assessment are improved.

[0080] Understandably, the rejection rate can indicate the rate at which content is refused to be answered or processed. A high rejection rate may indicate that there are problems with the content, such as sensitivity, controversy, or inappropriateness. Sentence fluency can be used to measure the fluency of sentence grammar and expression. The smoother the sentence, the more understandable and acceptable the text content is, thereby improving its security assessment. Description accuracy can reflect the consistency between the content description and the actual situation. High accuracy means that the information is clearly conveyed, which reduces misunderstandings and ambiguities and enhances the security of the content. Content security can directly evaluate the compliance and security of the content. High security means that the content is healthier, more compliant, and has lower risks.

[0081] In one embodiment of the present application, predicting the training sample subsets in different fields by using a preset rejection rate prediction model to obtain the rejection rates corresponding to the training sample subsets in different fields includes:

[0082] Preprocessing the training sample data in each of the training sample subsets to obtain a preprocessed training sample subset;

[0083] Based on the preprocessed training sample subset, the preset refusal rate prediction model is trained to obtain a refusal prediction result corresponding to each training sample data;

[0084] Based on the refusal to answer prediction result, the refusal rates corresponding to the training sample subsets in different fields are obtained.

[0085] It should be noted that the model will refuse to answer when the following situations occur: For example, it cannot handle or answer situations that are beyond its logical processing capabilities, knowledge depth, or data coverage; the input data or information is missing; or for ethical considerations, the question is vague or ambiguous.

[0086] Therefore, after obtaining the training sample subsets, the training sample data in each training sample subset can be preprocessed, for example, by removing duplicate content, missing content, unifying the format, etc., to obtain the preprocessed training sample data subset, and extract the features related to refusal to answer of each sample data, such as keywords, as the feature subset. Then, based on the training sample data subset and the feature subset, randomly select samples to iteratively train the preset refusal rate prediction model to obtain the refusal prediction result corresponding to each training sample data. Then, the refusal prediction result is statistically analyzed to obtain the refusal rate corresponding to the training sample data subset in each field. The answering ability of the model can be seen from multiple training sample subsets. For example, if the input is a historical and humanities text test set, the model's correct refusal rate is 60%. For example, if a value question is input, the model's correct refusal rate is 70%. In this way, the performance of the model facing sample sets in different fields can be examined.

[0087] It should be noted that the feature subset can be continuously updated, and features in the feature subset can be added or replaced according to each prediction result.

[0088] Furthermore, the preset rejection rate prediction model includes a plurality of decision trees, and the training of the preset rejection rate prediction model based on the preprocessed training sample subset includes:

[0089] Randomly selecting training sample data from the preprocessed training sample subset;

[0090] Inputting randomly selected training sample data into the decision tree to obtain the classification results output by each decision tree;

[0091] The classification result with the largest number of occurrences is selected from the classification results as the final classification result of the randomly selected training sample data.

[0092] Optionally, the preset refusal rate prediction model can be a random forest algorithm model. The random forest algorithm model can be composed of multiple decision trees, each of which can be trained on a training sample data subset and a feature subset, and each decision tree can output a prediction result for each sample data, and then select the prediction result with the most occurrences from all the prediction results as the final prediction result. After predicting each training sample data, all the final prediction results can be counted to obtain the number of refusals and answers, and then the refusal rate can be obtained based on the ratio between refusals and answers.

[0093] For example, there are T decision trees. For the input sample data x, each decision tree can give a classification result (x), then the final classification result of random forest (x) can be expressed by the following formula:

[0094] = ;

[0095] The final prediction result can be "answer" or "refuse to answer", and mode means taking the majority, that is, selecting the category with the most occurrences from the classification results of all decision trees as the final classification result. In the refusal rate model, this category is "answer" or "refuse to answer".

[0096] In an embodiment of the present application, the prediction of the different domain training sample subsets by using a preset sentence fluency evaluation model to obtain the sentence fluency corresponding to the different domain training sample subsets includes:

[0097] Preprocessing the training sample data in each of the training sample subsets to obtain a preprocessed training sample subset;

[0098] Based on the training sample subset, the preset sentence fluency model is trained to obtain a fluency prediction result corresponding to each training sample data;

[0099] Based on the fluency prediction result, the sentence fluency corresponding to the training sample subsets in different fields is obtained.

[0100] It should be noted that the fluency assessment model can automatically assess the fluency of a text. The model can analyze factors such as grammar, vocabulary collocation, and sentence structure in the text to make a judgment on the fluency of the text.

[0101] After obtaining the training sample subsets, the training sample data in each training sample subset can be preprocessed, such as word segmentation, stop word removal, part-of-speech tagging, etc., to obtain the preprocessed training sample subsets, and then the features related to fluency can be extracted from the preprocessed text to obtain the feature subsets, and then the preset fluency evaluation model can be trained based on the preprocessed training sample subsets and the feature subsets to obtain the fluency prediction results corresponding to each training sample data, and based on the fluency prediction results, the sentence fluency corresponding to the training sample subsets in different fields can be obtained. From multiple training sample subsets, it can be seen that the model's fluency evaluation of text data in different fields, for example, if the input is a historical and humanities text test set, the model fluency is 50%, and if the input is a value question, the model fluency is 80%. In this way, the performance of the model facing sample sets in different fields can be examined.

[0102] It should be noted that the feature subset can be continuously updated, and features in the feature subset can be added or replaced according to each prediction result.

[0103] Optionally, the preset fluency evaluation model may be a recurrent neural network model. During the forward propagation process of the recurrent neural network model, the output of the recurrent neural network layer may be captured. These outputs may be regarded as feature representations of text data. As required, the extracted features may be further processed, such as dimensionality reduction, standardization, etc. In text feature extraction, the recurrent neural network model may capture temporal dependencies in the text.

[0104] The forward propagation process of the recurrent neural network model can use the hyperbolic tangent function as the activation function, and the algorithm model is as follows:

[0105] ;

[0106] in, represents the input training sample sequence, represents the weight coefficient, Represents the bias vector.

[0107] Represents the hidden variable at time step t. Unlike the multi-layer perceptron, the hidden variable of the previous time step is saved here , and introduces a new weight coefficient to describe the use of hidden variables from the previous time step in the current time step.

[0108] The model of the output layer is as follows:

[0109] ;

[0110] in, represents the weight coefficient, represents the bias vector, That is the above , represents a hidden variable.

[0111] A softmax function can be added after the output layer to calculate the probability distribution of each category. The loss L can measure the distance between each output result o of each output layer and the training target y. The loss L is calculated internally and compared with the target y. The connection from the RNN input to the hidden is composed of the weight matrix U Parameterized, hidden-to-hidden recurrent connections are given by the weight matrix W The hidden-to-output connections are parameterized by the weight matrix V Parameterization, it should be noted that the weight matrix U、 Weight Matrix V With the weight matrix W These are existing parameters in the recurrent neural network model, which will not be described in detail in this application. The forward propagation based on this model is as follows:

[0112] ;

[0113] See also Figure 2 , at any sequence index t, the corresponding input is the corresponding sample sequence , which has a corresponding model prediction output , the actual output of the training sequence, i.e. the training target , and the loss function , plotting a recurrent neural network and its loss using recurrent connections, the same network is viewed as an unfolded computational graph where each node is now associated with a specific time instance.

[0114] Among them, the features related to fluency may include word frequency, part-of-speech distribution, dependency syntactic relationship, etc. They can reflect the characteristics of the text in terms of grammar, vocabulary collocation, etc.

[0115] In an embodiment of the present application, the prediction of the different field training sample subsets by a preset accuracy prediction model to obtain the description accuracy corresponding to the different field training sample subsets includes:

[0116] Preprocessing the training sample data in each of the training sample subsets to obtain a preprocessed training sample subset;

[0117] Performing a linear transformation on the training sample subset to obtain a query vector, a key vector, and a value vector;

[0118] Perform multi-head attention calculation based on the query vector, the key vector, and the value vector to obtain multiple prediction results;

[0119] The plurality of prediction results are concatenated, and the concatenated prediction results are linearly transformed to obtain a multi-head attention prediction result.

[0120] Based on the multi-head attention prediction results, the description accuracy corresponding to the training sample subsets in different fields is obtained.

[0121] After obtaining the training sample subsets, the training sample data in each training sample subset can be preprocessed to obtain the preprocessed training sample subsets, and then features related to the description accuracy can be extracted from the preprocessed text to obtain the feature subsets. Then, the preset accuracy assessment model can be trained based on the preprocessed training sample subsets and the feature subsets to obtain the description accuracy prediction results corresponding to each training sample data. Based on the accuracy prediction results, the description accuracy corresponding to the different training subsets can be obtained.

[0122] It should be noted that the feature subset can be continuously updated, and features in the feature subset can be added or replaced according to each prediction result.

[0123] Optionally, the preset accuracy prediction model can be a multi-head attention mechanism algorithm, which allows the model to simultaneously focus on information from different representation subspaces. Each head will independently learn different attention weights, and finally the outputs of these heads are concatenated and subjected to a linear transformation to obtain the final output result. Through the multi-head attention mechanism, the model can more comprehensively understand the input data and generate more accurate answers.

[0124] Specifically, the input preprocessed training sample data can be expressed as x={ , ,... }, the query (Q), key (K) and value (V) can be obtained through linear transformation, which can be obtained through the following formula:

[0125] ;

[0126] ;

[0127] ;

[0128] in, , ,and is a learnable weight matrix.

[0129] Based on the query (Q), key (K) and value (V), each head of the multi-head attention The attention score can be calculated by the following formula:

[0130] ;

[0131] in, represents the query vector, represents the key vector, is the dimension of the query and key, used to scale the dot product to prevent the gradient from being too large or too small.

[0132] The attention weight can then be obtained through the softmax function, as shown below:

[0133] ;

[0134] These weights are then used to perform a weighted sum of the value vectors to get the output for each head:

[0135] ;

[0136] in, Represents each head of the multi-head attention, represents weighted sum.

[0137] The outputs of all attention heads are concatenated and the final multi-head attention output is obtained through linear transformation, which can be specifically expressed as:

[0138] ;

[0139] Among them, Concat represents the concatenation operation. is the weight matrix of the output linear transformation.

[0140] In an embodiment of the present application, the predicting of the different domain training sample subsets by a preset content security prediction model to obtain the content security corresponding to the different domain training sample subsets includes:

[0141] Preprocessing the training sample data in each of the training sample subsets to obtain a preprocessed training sample subset;

[0142] Based on the linear regression model, linear regression processing is performed on the training sample subset to obtain an output result;

[0143] The output result is calculated by a preset activation function to obtain the content security corresponding to each training sample data;

[0144] Based on the content security corresponding to each training sample data, the content security corresponding to the training sample subsets in different fields is obtained.

[0145] After obtaining the training sample subsets, the training sample data in each training sample subset can be preprocessed to obtain a preprocessed training sample subset, and then features related to content security can be extracted from the preprocessed text to obtain a feature subset. Then, the preset content security prediction model can be trained based on the preprocessed training sample subset and the feature subset to obtain a prediction result of the content security corresponding to each training sample data. Based on the prediction result of the content security, the content security corresponding to the different training subsets can be obtained.

[0146] It should be noted that the feature subset can be continuously updated, and features in the feature subset can be added or replaced according to each prediction result.

[0147] Optionally, the preset content security prediction model can be a logistic regression algorithm model, which can identify and filter out unsafe or harmful content to help determine the possibility that the text content belongs to the unsafe category. It can be understood that the logistic regression algorithm model can distinguish between the two categories by finding a linear decision boundary. In content security detection, this boundary helps the model distinguish between safe and unsafe content.

[0148] Specifically, for the training sample data, a linear model can be constructed, which can be expressed as:

[0149] ;

[0150] in, is the weight parameter, is the input training sample data, and z is the output of the linear model.

[0151] Then, the sigmoid function can be used to convert the output z of linear regression into a probability value. This can be achieved through the following formula:

[0152] ;

[0153] The sigmoid function can map any real number to between 0 and 1 to represent probability;

[0154] If the probability is greater than a preset threshold, the text content can be considered unsafe, otherwise, the text content is considered safe.

[0155] The model parameter w can be trained by the maximum likelihood estimation algorithm so that the probability predicted by the model is as close as possible to the actual label. During the training process, w will be continuously adjusted to minimize the loss function. The model algorithm is as follows:

[0156] represents the set of samples of category c in the training set D, represents the parameter vector, then the parameter vector For the dataset The likelihood of can be expressed as:

[0157]

[0158] right Perform maximum likelihood estimation using Indicates that log likelihood can be used, and the algorithm model is as follows:

[0159] ;

[0160] Among them, argmax is used to find the category with the maximum predicted probability, and LL represents the log-likelihood function.

[0161] In an embodiment of the present application, preprocessing the training sample data in each of the training sample subsets to obtain the preprocessed training sample subsets includes:

[0162] Based on the requirements of each model, the training sample data in each of the training sample subsets is subjected to corresponding standardization processing;

[0163] Perform feature extraction on the standardized training sample data to obtain a feature subset;

[0164] Based on the feature subset, correspondingly labeling the training sample data;

[0165] Based on the labeled training sample data, the preprocessed training sample subset is generated.

[0166] Specifically, in response to the needs of the rejection rate prediction model, the fluency prediction model, the accuracy prediction model, and the content security prediction model, the training sample data in the training sample subset can be preprocessed accordingly. For example, for the fluency evaluation model, the training sample data can be preprocessed by word segmentation, stop word removal, part-of-speech tagging, etc. Then, feature extraction is performed on the preprocessed training sample data. For example, for the training sample data of the rejection rate prediction model, features related to rejection can be extracted, and for the training sample data of the fluency evaluation model, features related to fluency can be extracted. Then, the pre-trained sample data is labeled based on the feature, and each model is trained through the labeled training sample data.

[0167] In an embodiment of the present application, the security assessment result is obtained based on the rejection rate score, the sentence fluency score, the description accuracy score, and the content safety score, including:

[0168] Calculate a safety assessment score based on the refusal rate score, sentence fluency score, description accuracy score, and content safety score;

[0169] Based on the safety assessment score, the safety assessment result is determined.

[0170] Specifically, when the rejection rate score is obtained , Sentence Fluency Scoring , description accuracy score And content safety rating After that, it can be input into the pre-built security assessment index algorithm model as follows:

[0171] ;

[0172] Through the above formula, we can get the security evaluation index corresponding to the subsets of training samples in different fields. The security evaluation index can be a numerical interval, such as a value of 1-10. The closer to 10, the higher the security of the model. Otherwise, its security is low. Based on the security evaluation index, when the model is below the threshold, such as 5, the model can be optimized and updated.

[0173] It should be noted that They are the weight coefficients of refusal rate, sentence fluency, description accuracy and content security, which can be optimized through iterative training of training sample data to meet specific security assessment requirements.

[0174] It is understandable that the present application can perform multiple layers of evaluation. For the rejection rate prediction model, sentence fluency evaluation model, accuracy prediction model and content safety prediction model, the rejection rate, sentence fluency, accuracy and content safety can be evaluated respectively, and the rejection rate, sentence fluency, accuracy and content safety can also be comprehensively evaluated. The parameters of each model can be optimized and updated separately.

[0175] In the embodiments of the present application, a multi-dimensional security assessment analysis is performed based on the refusal rate, sentence fluency, description accuracy, and content security by combining machine learning, deep learning, and other technologies. The model has adaptive learning capabilities and can continuously update the assessment model through continuous learning and self-optimization to meet new challenges, thereby ensuring the timeliness and effectiveness of the assessment results and reducing the time and cost of manual participation.

[0176] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0177] In one embodiment, a security assessment device for generative artificial intelligence is provided, and the security assessment device for generative artificial intelligence corresponds one-to-one to the security assessment method for generative artificial intelligence in the above embodiment. Figure 3 As shown, the safety assessment device of the generative artificial intelligence includes a training sample data construction unit 10, a prediction unit 20, a weight allocation unit 30 and a safety assessment result determination unit 40. Each functional module is described in detail as follows:

[0178] A training sample data construction unit 10 is used to construct a training sample data set, wherein the training sample data set includes a plurality of training sample subsets in different fields;

[0179] The prediction unit 20 is used to perform prediction processing on the training sample subsets in different fields by using a preset prediction model to obtain the rejection rate, sentence fluency, description accuracy and content safety corresponding to the training sample subsets in different fields respectively;

[0180] A weight assignment unit 30 is used to assign corresponding weights to the rejection rate, sentence fluency, description accuracy and content safety based on the security assessment requirements, and calculate the rejection rate score, sentence fluency score, description accuracy score and content safety score based on the weights;

[0181] The security assessment result determination unit 40 is used to determine the security assessment result based on the rejection rate score, the sentence fluency score, the description accuracy score and the content safety score.

[0182] In an embodiment of the present application, the prediction unit 20 is further configured to:

[0183] Predicting the training sample subsets in different fields by using a preset rejection rate prediction model to obtain the rejection rates corresponding to the training sample subsets in different fields;

[0184] Predicting the subsets of training samples in different fields by using a preset sentence fluency evaluation model to obtain the sentence fluency corresponding to the subsets of training samples in different fields;

[0185] Predicting the subsets of training samples in different fields by using a preset accuracy prediction model to obtain the description accuracy corresponding to the subsets of training samples in different fields;

[0186] The different domain training sample subsets are predicted by a preset content security prediction model to obtain the content security corresponding to the different domain training sample subsets.

[0187] In an embodiment of the present application, the prediction unit 20 is further configured to:

[0188] Preprocessing the training sample data in each of the training sample subsets to obtain a preprocessed training sample subset;

[0189] Based on the preprocessed training sample subset, the preset refusal rate prediction model is trained to obtain a refusal prediction result corresponding to each training sample data;

[0190] Based on the refusal to answer prediction result, the refusal rates corresponding to the training sample subsets in different fields are obtained.

[0191] In one embodiment of the present application, the preset rejection rate prediction model includes multiple decision trees, and the prediction unit 20 is further used to:

[0192] Randomly selecting training sample data from the preprocessed training sample subset;

[0193] Inputting randomly selected training sample data into the decision tree to obtain the classification results output by each decision tree;

[0194] The classification result with the largest number of occurrences is selected from the classification results as the final classification result of the randomly selected training sample data.

[0195] In an embodiment of the present application, the prediction unit 20 is further configured to:

[0196] Preprocessing the training sample data in each of the training sample subsets to obtain a preprocessed training sample subset;

[0197] Based on the training sample subset, the preset sentence fluency model is trained to obtain a fluency prediction result corresponding to each training sample data;

[0198] Based on the fluency prediction result, the sentence fluency corresponding to the training sample subsets in different fields is obtained.

[0199] In an embodiment of the present application, the prediction unit 20 is further configured to:

[0200] Preprocessing the training sample data in each of the training sample subsets to obtain a preprocessed training sample subset;

[0201] Performing a linear transformation on the training sample subset to obtain a query vector, a key vector, and a value vector;

[0202] Perform multi-head attention calculation based on the query vector, the key vector, and the value vector to obtain multiple prediction results;

[0203] The plurality of prediction results are concatenated, and the concatenated prediction results are linearly transformed to obtain a multi-head attention prediction result.

[0204] Based on the multi-head attention prediction results, the description accuracy corresponding to the training sample subsets in different fields is obtained.

[0205] In an embodiment of the present application, the prediction unit 20 is further configured to:

[0206] Preprocessing the training sample data in each of the training sample subsets to obtain a preprocessed training sample subset;

[0207] Based on the linear regression model, linear regression processing is performed on the training sample subset to obtain an output result;

[0208] The output result is calculated by a preset activation function to obtain the content security corresponding to each training sample data;

[0209] Based on the content security corresponding to each training sample data, the content security corresponding to the training sample subsets in different fields is obtained.

[0210] In an embodiment of the present application, the prediction unit 20 is further configured to:

[0211] Based on the requirements of each model, the training sample data in each of the training sample subsets is subjected to corresponding standardization processing;

[0212] Perform feature extraction on the standardized training sample data to obtain a feature subset;

[0213] Based on the feature subset, correspondingly labeling the training sample data;

[0214] Based on the labeled training sample data, the preprocessed training sample subset is generated.

[0215] In one embodiment of the present application, the security assessment result determination unit 40 is further configured to:

[0216] Calculate a safety assessment score based on the refusal rate score, sentence fluency score, description accuracy score, and content safety score;

[0217] Based on the safety assessment score, the safety assessment result is determined.

[0218] In the embodiments of the present application, a multi-dimensional security assessment analysis is performed based on the refusal rate, sentence fluency, description accuracy, and content security by combining machine learning, deep learning, and other technologies. The model has adaptive learning capabilities and can continuously update the assessment model through continuous learning and self-optimization to meet new challenges, thereby ensuring the timeliness and effectiveness of the assessment results and reducing the time and cost of manual participation.

[0219] For the specific definition of the security assessment device for generative artificial intelligence, please refer to the definition of the security assessment method for generative artificial intelligence above, which will not be repeated here. Each module in the above-mentioned security assessment device for generative artificial intelligence can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0220] In one embodiment, a computer device is provided. The computer device may be a terminal device, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, and a network interface connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a readable storage medium. The readable storage medium stores computer-readable instructions. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer-readable instructions are executed by the processor, a security assessment method for generative artificial intelligence is implemented. The readable storage medium provided in this embodiment includes a non-volatile readable storage medium and a volatile readable storage medium.

[0221] In an embodiment of the present application, a computer device is provided, including a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, and when the processor executes the computer-readable instructions, the steps of the security assessment method of generative artificial intelligence as described above are implemented.

[0222] In an embodiment of the application, a readable storage medium is provided, wherein the readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the security assessment method of generative artificial intelligence as described above are implemented.

[0223] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing related hardware through computer-readable instructions, and the computer-readable instructions can be stored in a non-volatile readable storage medium or a volatile readable storage medium. When the computer-readable instructions are executed, they may include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0224] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0225] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A security assessment method for generative artificial intelligence, characterized in that: The method comprises: Constructing a training sample data set, wherein the training sample data set includes a plurality of training sample subsets in different fields; The prediction processing is performed on the training sample subsets in different fields through a preset prediction model to obtain the rejection rate, sentence fluency, description accuracy and content safety corresponding to the training sample subsets in different fields, respectively. The rejection rate is used to characterize the ratio of content that is rejected for answering or processing; the sentence fluency is used to measure the fluency of sentence grammar and expression; the description accuracy is used to reflect the consistency between the content description and the actual content; the content safety is used to directly evaluate the compliance and safety of the content; Based on the security assessment requirements, corresponding weights are assigned to the rejection rate, sentence fluency, description accuracy, and content safety, and based on the weights, a rejection rate score, a sentence fluency score, a description accuracy score, and a content safety score are calculated; Based on the refusal rate score, sentence fluency score, description accuracy score and content safety score, a security assessment index is calculated by a pre-built security assessment index algorithm model. When the security assessment index is greater than a preset threshold, it can be considered that the model meets the security assessment requirements, otherwise the weight needs to be adjusted to optimize the model; The predicting process is performed on the training sample subsets in different fields by using a preset prediction model to obtain the rejection rate, sentence fluency, description accuracy and content safety corresponding to the training sample subsets in different fields, including: Predicting the training sample subsets in different fields by using a preset rejection rate prediction model to obtain the rejection rates corresponding to the training sample subsets in different fields; Predicting the subsets of training samples in different fields by using a preset sentence fluency evaluation model to obtain the sentence fluency corresponding to the subsets of training samples in different fields; Predicting the subsets of training samples in different fields by using a preset accuracy prediction model to obtain the description accuracy corresponding to the subsets of training samples in different fields; The different domain training sample subsets are predicted by a preset content security prediction model to obtain the content security corresponding to the different domain training sample subsets.

2. The security assessment method of generative artificial intelligence according to claim 1, characterized in that: The predicting of the training sample subsets in different fields by using a preset rejection rate prediction model to obtain the rejection rates corresponding to the training sample subsets in different fields includes: Preprocessing the training sample data in each of the training sample subsets to obtain a preprocessed training sample subset; Based on the preprocessed training sample subset, the preset refusal rate prediction model is trained to obtain a refusal prediction result corresponding to each training sample data; Based on the refusal to answer prediction result, the refusal rates corresponding to the training sample subsets in different fields are obtained.

3. The security assessment method of generative artificial intelligence according to claim 2, characterized in that: The preset rejection rate prediction model includes a plurality of decision trees, and the training of the preset rejection rate prediction model based on the preprocessed training sample subset includes: Randomly selecting training sample data from the preprocessed training sample subset; Inputting randomly selected training sample data into the decision tree to obtain the classification results output by each decision tree; The classification result with the largest number of occurrences is selected from the classification results as the final classification result of the randomly selected training sample data.

4. The security assessment method of generative artificial intelligence according to claim 1, characterized in that: The predicting of the different domain training sample subsets by using a preset sentence fluency evaluation model to obtain the sentence fluency corresponding to the different domain training sample subsets includes: Preprocessing the training sample data in each of the training sample subsets to obtain a preprocessed training sample subset; Based on the training sample subset, the preset sentence fluency evaluation model is trained to obtain a fluency prediction result corresponding to each training sample data; Based on the fluency prediction result, the sentence fluency corresponding to the training sample subsets in different fields is obtained.

5. The security assessment method of generative artificial intelligence according to claim 1, characterized in that: The predicting of the different domain training sample subsets by using a preset accuracy prediction model to obtain the description accuracy corresponding to the different domain training sample subsets includes: Preprocessing the training sample data in each of the training sample subsets to obtain a preprocessed training sample subset; Performing a linear transformation on the training sample subset to obtain a query vector, a key vector, and a value vector; Perform multi-head attention calculation based on the query vector, the key vector, and the value vector to obtain multiple prediction results; The plurality of prediction results are concatenated, and the concatenated prediction results are linearly transformed to obtain a multi-head attention prediction result. Based on the multi-head attention prediction results, the description accuracy corresponding to the training sample subsets in different fields is obtained.

6. The security assessment method of generative artificial intelligence according to claim 1, characterized in that: The step of predicting the different domain training sample subsets by using a preset content security prediction model to obtain content security corresponding to the different domain training sample subsets includes: Preprocessing the training sample data in each of the training sample subsets to obtain a preprocessed training sample subset; Based on the linear regression model, linear regression processing is performed on the training sample subset to obtain an output result; The output result is calculated by a preset activation function to obtain the content security corresponding to each training sample data; Based on the content security corresponding to each training sample data, the content security corresponding to the training sample subsets in different fields is obtained.

7. The security assessment method for generative artificial intelligence according to any one of claims 2 to 6, characterized in that: The preprocessing of the training sample data in each of the training sample subsets to obtain the preprocessed training sample subsets includes: Based on the requirements of each model, the training sample data in each of the training sample subsets is subjected to corresponding standardization processing; Perform feature extraction on the standardized training sample data to obtain a feature subset; Based on the feature subset, correspondingly labeling the training sample data; Based on the labeled training sample data, the preprocessed training sample subset is generated.

8. A security assessment device for generative artificial intelligence, characterized in that: The device comprises: A training sample data construction unit, used to construct a training sample data set, wherein the training sample data set includes a plurality of training sample subsets in different fields; A prediction unit is used to perform prediction processing on the training sample subsets in different fields through a preset prediction model to obtain the rejection rate, sentence fluency, description accuracy and content safety corresponding to the training sample subsets in different fields, respectively. The rejection rate is used to characterize the ratio of content that is rejected for answering or processing; the sentence fluency is used to measure the fluency of sentence grammar and expression; the description accuracy is used to reflect the consistency between the content description and the actual content; the content safety is used to directly evaluate the compliance and safety of the content; A weight allocation unit, configured to allocate corresponding weights to the rejection rate, sentence fluency, description accuracy, and content safety based on security assessment requirements, and calculate a rejection rate score, a sentence fluency score, a description accuracy score, and a content safety score based on the weights; A security assessment result determination unit is used to calculate a security assessment index based on the refusal rate score, the sentence fluency score, the description accuracy score and the content safety score through a pre-built security assessment index algorithm model. When the security assessment index is greater than a preset threshold, it can be considered that the model meets the security assessment requirements, otherwise the weight needs to be adjusted to optimize the model; The prediction unit is further used for: Predicting the training sample subsets in different fields by using a preset rejection rate prediction model to obtain the rejection rates corresponding to the training sample subsets in different fields; Predicting the subsets of training samples in different fields by using a preset sentence fluency evaluation model to obtain the sentence fluency corresponding to the subsets of training samples in different fields; Predicting the subsets of training samples in different fields by using a preset accuracy prediction model to obtain the description accuracy corresponding to the subsets of training samples in different fields; The different domain training sample subsets are predicted by a preset content security prediction model to obtain the content security corresponding to the different domain training sample subsets.

Images