Abnormality detection method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, in particular to an anomaly detection method and device, electronic equipment and a storage medium, and is used for improving the detection accuracy of anomaly input. The method comprises the steps of obtaining a to-be-detected input text; based on the text features of the input text, determining a classification result of the input text, and when the classification result represents that the input text is suspicious input, extracting suspicious fragments in the input text; and determining an anomaly detection result of the input text based on a target similarity between the suspicious fragment and each abnormal word contained in the blacklist. Therefore, the generalization of the blacklist can be improved, so that the interception of abnormal input of deformation and modification words is improved, the false drop rate is reduced, an input object is more efficiently and effectively prevented from using an abnormal input text, and a text generation image model is induced to generate harmful image content.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an anomaly detection method, device, electronic device and storage medium. Background Art

[0002] With the development of artificial intelligence technology, multimodal large models have gradually become a hot spot for research and application in the field of artificial intelligence. The text-to-image generation in the multimodal large model application allows users to easily use the text-to-image model to generate high-quality and realistic images, bringing great convenience to people's lives and work. At the same time, it may also be used by people to create abnormal images containing harmful content, bringing risks of use.

[0003] In order to prevent the text-generated image model from being abused and used to create abnormal images, in the prior art, it is usually detected whether the input text contains abnormal words in a preset blacklist, determine whether the input text is abnormal input, and prohibit the abnormal input from entering the text-generated image model.

[0004] However, the above method is limited to the coverage of the blacklist, and the input object can change the expression of abnormal words in the input text, thereby bypassing the blacklist check, resulting in missed detection, low detection accuracy of abnormal input, and poor security protection effect of the text-to-image model. Summary of the invention

[0005] Embodiments of the present application provide an abnormality detection method, device, electronic device, and storage medium to improve the detection accuracy of abnormal input.

[0006] The specific technical solutions provided by the embodiments of this application are as follows:

[0007] In a first aspect, a method for detecting anomalies is provided, comprising:

[0008] Get the input text to be detected;

[0009] Determine a classification result of the input text based on text features of the input text, and extract suspicious segments from the input text when the classification result indicates that the input text is suspicious input;

[0010] Based on the target similarity between the suspicious segment and each abnormal word included in the blacklist, the anomaly detection result of the input text is determined.

[0011] In a possible embodiment, based on the text features of the input text, determining the classification result of the input text, and when the classification result characterizes that the input text is suspicious input, extracting the suspicious segment in the input text includes:

[0012] Extracting initial text features of input text;

[0013] Based on the attention weight corresponding to the initial text feature, the attention score of each initial text feature element contained in the initial text feature is obtained;

[0014] Based on the obtained attention scores, the initial text features are updated to obtain the target text features, and based on the target text features, the classification results of the input text are obtained;

[0015] When the classification result characterizes the input text as suspicious input, a suspicious segment in the input text is determined based on each attention score.

[0016] Through the above method, the classification accuracy and recognition efficiency can be improved, and suspicious segments can also be extracted.

[0017] In a possible embodiment, the classification result and the suspicious segment are obtained by inputting the input text into a target classification model, wherein the target classification model is trained in the following manner:

[0018] Based on the sample input text set, the classification model to be trained is iteratively trained to obtain the target classification model, wherein the sample input text set includes abnormal sample input text and normal sample input text. In one iterative training process, the following operations are performed:

[0019] Extracting sample initial text features of the selected sample input text;

[0020] Based on the sample attention weight corresponding to the sample initial text feature, obtain the sample attention score of each sample initial text feature element contained in the sample initial text feature;

[0021] Based on the attention scores of each sample obtained, the initial text features of the sample are updated to obtain the sample target text features, and based on the sample target text features, the classification prediction results of the sample input text are obtained, and the parameters are adjusted based on the loss values ​​corresponding to the classification prediction results.

[0022] Through the above method, the accuracy of the target classification model can be improved.

[0023] In a possible embodiment, determining an anomaly detection result of the input text based on the target similarity between the suspicious segment and each abnormal word included in the blacklist includes:

[0024] Vectorize the suspicious fragment to obtain the vector to be detected;

[0025] Calculate the cosine similarity between the abnormal vectors of each abnormal word included in the blacklist and the vector to be detected, and obtain the target similarity between each abnormal word and the suspicious segment;

[0026] Based on the obtained similarities of each target, an anomaly detection result of the input text is determined.

[0027] Through the above method, by calculating the cosine similarity between vectors, the target similarity of suspicious fragments and abnormal words is calculated, which ensures the accuracy of target similarity calculation and improves the accuracy of detection.

[0028] In a possible embodiment, determining an anomaly detection result of the input text based on the obtained similarities of each target includes:

[0029] If there is an abnormal word among the abnormal words whose target similarity with the suspicious segment is greater than a preset similarity threshold, the input text is determined to be an abnormal input;

[0030] If there is no abnormal word among the abnormal words whose target similarity with the suspicious segment is greater than the similarity threshold, the input text is determined to be normal input.

[0031] In a possible embodiment, the method further includes:

[0032] When the classification result indicates that the input text is normal input, the input text is input into the image generation model to obtain a target image corresponding to the input text.

[0033] In a possible embodiment, after determining the abnormality detection result of the input text based on the target similarity between the suspicious segment and each abnormal word included in the blacklist, the method further includes:

[0034] When the anomaly detection result indicates that the input text is normal input, the input text is input into the image generation model to obtain a target image corresponding to the input text;

[0035] When the anomaly detection result indicates that the input text is an abnormal input, an abnormal prompt information is returned, and the abnormal prompt information indicates that the input text is an abnormal input.

[0036] Through the above method, the detected abnormal input is intercepted to prevent the multimodal model from being maliciously used to generate abnormal images, thereby effectively improving the security of the text-to-image model.

[0037] In a second aspect, the present application provides an abnormality detection device, the device comprising:

[0038] An acquisition module is used to acquire input text to be detected;

[0039] A first detection module is used to determine a classification result of the input text based on the text features of the input text, and when the classification result indicates that the input text is suspicious input, extract a suspicious segment in the input text;

[0040] The second detection module is used to determine anomaly detection results of the input text based on the target similarity between the suspicious segment and each abnormal word included in the blacklist.

[0041] In a possible embodiment, based on the text features of the input text, a classification result of the input text is determined, and when the classification result characterizes that the input text is suspicious input, when extracting suspicious segments in the input text, the first detection module is further used to:

[0042] Extracting initial text features of input text;

[0043] Based on the attention weight corresponding to the initial text feature, the attention score of each initial text feature element contained in the initial text feature is obtained;

[0044] Based on the obtained attention scores, the initial text features are updated to obtain the target text features, and based on the target text features, the classification results of the input text are obtained;

[0045] When the classification result characterizes the input text as suspicious input, a suspicious segment in the input text is determined based on each attention score.

[0046] In a possible embodiment, the classification result and the suspicious segment are obtained by inputting the input text into a target classification model, and the apparatus further includes a training module, which is used to:

[0047] Based on the sample input text set, the classification model to be trained is iteratively trained to obtain the target classification model, wherein the sample input text set includes abnormal sample input text and normal sample input text. In one iterative training process, the following operations are performed:

[0048] Extracting sample initial text features of the selected sample input text;

[0049] Based on the sample attention weight corresponding to the sample initial text feature, obtain the sample attention score of each sample initial text feature element contained in the sample initial text feature;

[0050] Based on the attention scores of each sample obtained, the initial text features of the sample are updated to obtain the sample target text features, and based on the sample target text features, the classification prediction results of the sample input text are obtained, and the parameters are adjusted based on the loss values ​​corresponding to the classification prediction results.

[0051] In a possible embodiment, when determining the abnormality detection result of the input text based on the target similarity between the suspicious segment and each abnormal word included in the blacklist, the second detection module is further used to:

[0052] Vectorize the suspicious fragment to obtain the vector to be detected;

[0053] Calculate the cosine similarity between the abnormal vectors of each abnormal word included in the blacklist and the vector to be detected, and obtain the target similarity between each abnormal word and the suspicious segment;

[0054] Based on the obtained similarities of each target, an anomaly detection result of the input text is determined.

[0055] In a possible embodiment, when determining the abnormality detection result of the input text based on the obtained similarities of each target, the second detection module is further used to:

[0056] If there is an abnormal word among the abnormal words whose target similarity with the suspicious segment is greater than a preset similarity threshold, the input text is determined to be an abnormal input;

[0057] If there is no abnormal word among the abnormal words whose target similarity with the suspicious segment is greater than the similarity threshold, the input text is determined to be normal input.

[0058] In a possible embodiment, the second detection module is further used for:

[0059] When the classification result indicates that the input text is normal input, the input text is input into the image generation model to obtain a target image corresponding to the input text.

[0060] In a possible embodiment, after determining the abnormality detection result of the input text based on the target similarity between the suspicious segment and each abnormal word included in the blacklist, the second detection module is further used to:

[0061] When the anomaly detection result indicates that the input text is normal input, the input text is input into the image generation model to obtain a target image corresponding to the input text;

[0062] When the anomaly detection result indicates that the input text is an abnormal input, an abnormal prompt information is returned, and the abnormal prompt information indicates that the input text is an abnormal input.

[0063] According to a third aspect, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any one of the methods described in the first aspect are implemented.

[0064] According to a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method described in any one of the first aspects are implemented.

[0065] In a fifth aspect, the present application provides a computer program product, comprising: a computer program code, when the computer program code is run on a computer, the computer executes any one of the methods described in the first aspect.

[0066] In the embodiment of the present application, the input text to be detected is obtained, and then the classification result of the input text is determined based on the text features of the input text, and when the classification result characterizes the input text as suspicious input, the suspicious fragments in the input text are extracted, and finally the abnormal detection result of the input text is determined based on the target similarity between the suspicious fragments and each abnormal word included in the blacklist. In this way, the suspicious input is first filtered, and the suspicious fragments in the suspicious input are extracted, and then the similarity between the suspicious fragments and each abnormal word is calculated, which can improve the generalization of the blacklist, thereby improving the interception of abnormal input with deformation and modified wording, and reducing the false detection rate, more efficiently and effectively preventing the input object from using abnormal input text, inducing the text generation image model to generate harmful image content, and reducing the risk of malicious abuse of the text generation image model. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 A schematic diagram of possible application scenarios in the embodiments of the present application;

[0068] Figure 2 A schematic diagram of a flow chart of an iterative training of a training target classification model in an embodiment of the present application;

[0069] Figure 3 This is a first schematic diagram of the self-attention mechanism in an embodiment of the present application;

[0070] Figure 4 This is a second schematic diagram of the self-attention mechanism in an embodiment of the present application;

[0071] Figure 5 A flowchart of an abnormality detection method in an embodiment of the present application;

[0072] Figure 6 Another flowchart of an abnormality detection method in an embodiment of the present application is shown;

[0073] Figure 7 This is a schematic diagram of the structure of an abnormality detection device in an embodiment of the present application;

[0074] Figure 8 Schematic diagram of the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0075] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0076] To facilitate understanding of the technical solutions provided in the embodiments of the present application, some key terms used in the embodiments of the present application are explained here:

[0077] Multimodal big model: an artificial intelligence model that can process multiple data types (for example, text, images, and videos). This type of model provides more comprehensive and accurate information by combining different types of data.

[0078] Text-to-image (T2I) model: is a machine learning model that combines a language understanding model and an image generation model, using a natural language description as input and generating an image that matches the description.

[0079] Semantic vector model (embedding): A pre-trained model trained on large-scale text data with strong semantic representation capabilities. It can convert natural language into vectors and use the "distance" between vectors to measure the similarity between data samples.

[0080] Self-attention mechanism: The main idea is to calculate the correlation between input data, and use this correlation to re-encode the input data. The basic form of the input data is a group of elements.

[0081] The following is a brief introduction to the design concept of the embodiment of the present application:

[0082] At present, with the development of artificial intelligence technology, especially since the launch of large language models, there has been an explosive growth in artificial intelligence innovation. Multimodal large models have also become a hot spot for research and application in the field of artificial intelligence because they can understand and generate multiple types of data, conduct richer and deeper understanding, and thus provide richer and more accurate information. The text-generated image in the multimodal large model application allows users to easily use the text-generated image model to generate high-quality and realistic images, which brings great convenience to people's lives and work. At the same time, it may also be used by people to create abnormal images containing harmful content (for example, malicious defamation of public figures, sex, harassment, and illegal activities), which brings risks of use. In order to prevent the text-generated image model from being abused and used to create abnormal images, it is necessary to add a safety protection mechanism during the generation process to allow the model to provide services to people more healthily and compliantly.

[0083] In the prior art, security protection mechanisms usually adopt the following two methods:

[0084] Method 1: Based on the classification model, detect whether the generated output image is compliant to prevent abnormal images from being generated and returned to the user.

[0085] When using method 1, it relies on the training data of the classification model. When expanding new abnormal categories, the model needs to be retrained, which is not convenient for expansion. In addition, the abnormal images have been generated, wasting computing resources.

[0086] Method 2: Detect whether the input text contains abnormal words in the preset blacklist, determine whether the input text is abnormal input, and prohibit abnormal input from entering the text generation image model.

[0087] When using method 2, it is limited to the coverage of the blacklist, and the input object can change the expression of abnormal words in the input text, thereby bypassing the blacklist check, resulting in missed detection, low detection accuracy of abnormal input, and poor security protection effect of the text-to-image model.

[0088] In view of this, in the embodiments of the present application, an anomaly detection method, device, electronic device and storage medium are proposed. The input text to be detected is obtained, and then the classification result of the input text is determined based on the text features of the input text, and when the classification result characterizes the input text as a suspicious input, the suspicious fragment in the input text is extracted, and then the target similarity between the suspicious fragment and each abnormal word included in the blacklist is determined. The abnormal detection result of the input text. In this way, the suspicious input is first filtered, and the suspicious fragment in the suspicious input is extracted, and then the similarity between the suspicious fragment and each abnormal word is calculated, which can improve the generalization of the blacklist, thereby improving the interception of abnormal input with deformation and modified wording, and reducing the false detection rate, more efficiently and effectively preventing the input object from using abnormal input text, inducing the text generation image model to generate harmful image content, and reducing the risk of malicious abuse of the text generation image model. In addition, it also avoids the waste of computing resources of the text generation image model for abnormal input text.

[0089] The preferred implementation modes of the present application are described in detail below with reference to the accompanying drawings.

[0090] See also Figure 1 , which is a schematic diagram of possible application scenarios in the embodiments of the present application. The application scenario diagram includes a server 110 and terminal devices 120 (including terminal devices 1201, 1202, ..., 120n).

[0091] The server 110 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal device 120 and the server 110 may be directly or indirectly connected via wired or wireless communication, which is not limited in this application.

[0092] The terminal device 120 includes but is not limited to mobile phones, tablet computers, laptops, desktop computers, e-book readers, intelligent voice interaction devices, smart home appliances, car terminals and other devices; various software can be installed on the terminal device, such as applications and applets.

[0093] It should be noted that Figure 1 The figure is only an example. In fact, the number of terminal devices 120 and servers 110 is not limited and is not specifically limited in the embodiments of the present application.

[0094] In the embodiment of the present application, based on the sample input text set, the classification model to be trained is iteratively trained to obtain a target classification model.

[0095] Among them, the sample input text set includes abnormal sample input text and normal sample input text, and the classification model to be trained can be a model of transformer structure, such as the pre-trained model Roberta on the Chinese corpus, etc., which is not limited in the embodiments of the present application.

[0096] The transformer structure is essentially a stack of encoder-decoder architectures. Each encoder contains a self-attention layer and a feedforward neural network layer. The core idea is to add a self-attention mechanism so that when the encoder encodes a specific word, it not only uses the word, but also uses the information of other words in the input sentence, aiming to enhance the semantic representation of the target word. The decoder also contains two layers of encoders, and there is an attention layer between the two layers. It gives higher weight to the important semantic parts of the sentence and lower weight to the unimportant parts, so that the decoder pays more attention to the important information parts in the sentence. In particular, the use of the multi-head attention mechanism allows the encoder and decoder to consider the different ways of integrating the target word with other words in the sentence in various semantic scenarios, and finally outputs the enhanced representation of the target word after integrating the full text information.

[0097] See also Figure 2 As shown, it is a schematic diagram of a flow chart of an iterative training of a training target classification model in an embodiment of the present application. Figure 2, the specific operations performed in one iteration of training are described in detail:

[0098] Step 20: Extract sample initial text features of the selected sample input text.

[0099] In the embodiment of the present application, sample input text for this iterative training is selected from a sample input text set, and sample initial text features of the sample input text are extracted through an encoder.

[0100] Step 21: Based on the sample attention weight corresponding to the sample initial text feature, obtain the sample attention score of each sample initial text feature element contained in the sample initial text feature.

[0101] Among them, the sample attention weight includes query weight, key weight and value weight.

[0102] In an embodiment of the present application, matrix multiplication is performed on the sample initial text features with each sample query weight matrix and each sample key weight matrix to generate corresponding sample query matrices and sample key matrices, and then matrix multiplication is performed on each sample query matrix and the transpose of the corresponding sample key matrix, and the matrix is ​​divided by the dimension size of the sample key matrix to obtain each sample attention score matrix, and the normalized sample attention score matrix is ​​obtained using the softmax function, that is, the sample attention score of each sample initial text feature element contained in the sample initial text features is obtained.

[0103] For example, see Figure 3 As shown in the figure, it is a first schematic diagram of the self-attention mechanism in the embodiment of the present application, assuming that the weight matrix of each sample query is W i Q , the weight matrix of each sample key is W i K , i takes 0-7, and the sample initial text feature X is respectively combined with each sample query weight matrix W i Q And each sample key weight matrix W i K Perform matrix multiplication to generate query matrices Q for each sample i And each sample key matrix K i , and then query the matrix Q for each sample i And the corresponding sample key matrix K i T Multiply the transpose of and divide by the dimension size of the sample key matrix under the square root to obtain the attention score matrix of each sample Then normalize the attention score matrix of each sample, that is, softmax = A i , where d k is the sample key matrix K i The dimension size of .

[0104] Step 22: Based on the attention scores of each sample obtained, the initial text features of the sample are updated to obtain the sample target text features, and based on the sample target text features, the classification prediction results of the sample input text are obtained, and the parameters are adjusted based on the loss values ​​corresponding to the classification prediction results.

[0105] In an embodiment of the present application, matrix multiplication is performed on the initial text features of the samples and the weight matrices of each sample value to generate a corresponding sample value matrix, and then the normalized attention score matrix of each sample is multiplied by the corresponding sample value matrix to obtain the intermediate text features of each sample. Finally, the intermediate text features of each sample are concatenated and multiplied with the output matrix to obtain the sample target text features, and then based on the sample target text features, the classification prediction results of the sample input text are obtained, and the parameters are adjusted based on the loss value corresponding to the classification prediction results.

[0106] For example, Figure 3 As shown, assuming that the weight matrix of each sample value is W i V , the sample initial text feature X is respectively combined with each sample value weight matrix W i V Perform matrix multiplication to generate the sample value matrix V i , and then the normalized sample attention score matrix and the corresponding sample value matrix V i Multiply them together to obtain the intermediate text features H of each sample i Finally, the intermediate text features of each sample are concatenated and combined with the output matrix W O Multiply them together to obtain the sample target text feature Z.

[0107] For example, see Figure 4 As shown, it is a second schematic diagram of the self-attention mechanism in the embodiment of the present application. In the embodiment of the present application, based on a set of attention weights, the sample intermediate text features corresponding to the set of attention weights are obtained. The set of attention weights includes a sample query weight matrix W Q , a sample key weight matrix W K and a sample value weight matrix W V , the sample initial text features X are respectively combined with the sample query weight matrix W Q , sample key weight matrix W K Perform matrix multiplication W with the sample value weight matrix V , generate the sample query matrix Q, sample key matrix K1 and sample value matrix V, and then transpose the sample query matrix Q1 and the sample key matrix K T Do matrix multiplication and divide by the dimension size of the sample key matrix under the square root Obtain the sample attention score matrix A, then use the softmax function to obtain the normalized sample attention score matrix, and finally multiply the normalized sample attention score matrix softmax(A) with the sample value matrix V to obtain the intermediate text features H of each sample.

[0108] Based on the above embodiments, see Figure 5 FIG. 1 is a flow chart of an abnormality detection method in an embodiment of the present application, which specifically includes:

[0109] Step 50: Obtain input text to be detected.

[0110] In an embodiment of the present application, after the input object triggers an input operation of generating an image from text on a terminal device, the server obtains the input text input by the input object in response to the image generation request triggered by the input object as the input text to be detected.

[0111] Step 51: Based on the text features of the input text, determine the classification result of the input text, and when the classification result indicates that the input text is suspicious input, extract the suspicious segments in the input text.

[0112] Among them, the suspicious segment is composed of one suspicious element or multiple suspicious elements, which is not limited in the embodiment of the present application. The suspicious element is a word or a short sentence suspected of harmful content.

[0113] In the embodiment of the present application, a possible embodiment is provided to determine the classification result of the input text based on the text features of the input text, and to extract the suspicious segments in the input text when the classification result characterizes the input text as suspicious input. The specific steps are as follows:

[0114] Step 510: Extract initial text features of the input text.

[0115] In the embodiment of the present application, the initial text features of the input text are extracted through the trained target classification model.

[0116] Step 511: Based on the attention weight corresponding to the initial text feature, obtain the attention score of each initial text feature element included in the initial text feature.

[0117] In an embodiment of the present application, the initial text features are matrix multiplied with each query weight matrix and each key weight matrix to generate corresponding query matrices and key matrices, and then each query matrix is ​​matrix multiplied with the transpose of the corresponding key matrix and divided by the dimension size of the key matrix to obtain each attention score matrix, and the softmax function is used to obtain each normalized attention score matrix, that is, the attention score of each initial text feature element contained in the initial text feature is obtained.

[0118] Step 512: Based on the obtained attention scores, the initial text features are updated to obtain the target text features, and based on the target text features, the classification results of the input text are obtained.

[0119] In an embodiment of the present application, the initial text features are matrix multiplied with each value weight matrix to generate a corresponding value matrix, and then the normalized attention score matrices are multiplied with the corresponding value matrix to obtain each intermediate text feature. Finally, the intermediate text features are concatenated and multiplied with the output matrix to obtain the target text features, and then based on the target text features, the classification prediction results of the input text are obtained.

[0120] Step 513: When the classification result characterizes the input text as suspicious input, based on each attention score, determine the suspicious segment in the input text.

[0121] In an embodiment of the present application, after obtaining the classification result, when the classification result characterizes the input text as suspicious input, based on the attention score of each element in the input text and combined with the suspicious conditions, the suspicious fragments in the input text are determined.

[0122] Among them, the suspicious condition can be that the attention score is greater than the attention score threshold, or it can be the top N attention scores after the attention scores are sorted, which is not limited in the embodiments of the present application.

[0123] For example, determine M suspicious elements among the elements whose attention scores are greater than the attention score threshold, and the M suspicious elements constitute a suspicious fragment.

[0124] For another example, the top three elements in terms of attention scores are grouped into a suspicious segment.

[0125] In this way, by including a target classification model with a self-attention mechanism, the recognition efficiency of classification accuracy can be improved, and suspicious fragments can also be extracted.

[0126] In addition, it is worth noting that, in the embodiment of the present application, when the classification result indicates that the input text is normal input, the input text is input into the image generation model (diffusion model) to obtain a target image corresponding to the input text.

[0127] For example, assuming that the input text A is: "red apple", the target classification model recognizes the input text A as normal input, inputs the input text A into the image generation model, and obtains a target image containing a red apple.

[0128] Step 52: Determine anomaly detection results of the input text based on the target similarity between the suspicious segment and each abnormal word included in the blacklist.

[0129] Among them, the blacklist is pre-set, and the anomaly detection result represents whether the input text is an abnormal input.

[0130] In the embodiment of the present application, the suspicious segment is used as an input parameter and input into the target detection model for anomaly detection to obtain anomaly detection results of the input text.

[0131] The target detection model may be an embedding, for example, bge-large-zh-v1.5 on huggingface is used as the target detection model.

[0132] In this way, since the input text is usually long and contains suspicious fragments, directly calculating the similarity between the input text and the blacklist has a low accuracy. Therefore, by calculating the similarity between the suspicious fragments of the input text and the abnormal words in the blacklist, the accuracy of the similarity calculation is guaranteed, thereby improving the accuracy of detection.

[0133] Specifically, in an embodiment of the present application, a possible embodiment is provided to determine anomaly detection results of input text based on target similarity between suspicious segments and each abnormal word included in the blacklist, and the specific steps are as follows:

[0134] Step 520: vectorize the suspicious segment to obtain a vector to be detected.

[0135] In an embodiment of the present application, a target detection model is used to vectorize the suspicious segment to obtain a vector to be detected corresponding to the suspicious segment, that is, the suspicious segment is mapped to the latent space of the target detection model to obtain the vector to be detected.

[0136] Step 521: Calculate the cosine similarity between the abnormal vectors of each abnormal word included in the blacklist and the vector to be detected, and obtain the target similarity between each abnormal word and the suspicious segment.

[0137] In the embodiment of the present application, for each abnormal word included in the blacklist, the following operations are performed respectively: the cosine similarity between an abnormal vector corresponding to an abnormal word and the vector to be detected is calculated, and the cosine similarity is used as the target similarity between the abnormal word and the suspicious segment.

[0138] For example, if the cosine similarity between the abnormal vector corresponding to the abnormal word A and the vector to be detected is 0.78, then the target similarity between the abnormal word A and the suspicious segment is 0.78.

[0139] In this way, by calculating the cosine similarity between vectors, the target similarity of suspicious segments and abnormal words is calculated, which ensures the accuracy of target similarity calculation.

[0140] Step 522: Based on the obtained similarities of each target, determine the abnormality detection result of the input text.

[0141] In the embodiment of the present application, if the similarities of each target meet the preset abnormal conditions, the abnormal detection result of the input text is determined.

[0142] Specifically, in the embodiment of the present application, a possible embodiment is provided to determine the abnormality detection result of the input text based on the obtained similarities of each target, and the specific steps are as follows:

[0143] Step 5220: Determine whether there is an abnormal word among the abnormal words whose target similarity with the suspicious segment is greater than a preset similarity threshold. If so, execute step 5221; otherwise, execute step 5222.

[0144] Step 5221: Determine that the input text is abnormal input.

[0145] In the embodiment of the present application, if there is an abnormal word among the abnormal words whose target similarity with the suspicious segment is greater than a preset similarity threshold, the input text is determined to be an abnormal input.

[0146] For example, assuming that the preset similarity threshold is 0.75, the abnormal words include: abnormal word A, abnormal word B, abnormal word C, the target similarity between abnormal word A and the suspicious segment is 0.78, the target similarity between abnormal word B and the suspicious segment is 0.5, the target similarity between abnormal word C and the suspicious segment is 0.5, and the target similarity between abnormal word A and the suspicious segment, 0.78, is greater than the preset similarity threshold of 0.75, then the input text is determined to be abnormal input.

[0147] In this way, by comparing the similarity between the suspicious fragment and the abnormal words in the blacklist, the generalization of the blacklist is improved, the deformed blacklist is detected, and the interception of abnormal input with deformed and modified wording is improved.

[0148] Further, in an embodiment of the present application, after obtaining an abnormality detection result of the input text, when the abnormality detection result indicates that the input text is abnormal input, abnormality prompt information is returned, and the abnormality prompt information indicates that the input text is abnormal input.

[0149] For example, suppose the input text B is: "A person lying on the ground, with a lot of red liquid around him, 8k, crime scene photo", the target classification model identifies the input text B as a suspicious input, and extracts the suspicious elements "a lot of red liquid around him" and "crime scene", then concatenates the suspicious elements, and inputs the suspicious segment corresponding to the text B. The similarity between the suspicious segment and the abnormal word "murder" included in the blacklist is greater than the similarity threshold, thereby intercepting the input text B from entering the image generation model and directly returning the abnormal prompt message "Your input contains abnormalities".

[0150] In this way, the detected abnormal input is intercepted to prevent the multimodal model from being maliciously exploited to generate abnormal images, which effectively improves the security of the text-to-image model.

[0151] Step 5221: Determine that the input text is normal input.

[0152] In the embodiment of the present application, if there is no abnormal word among the abnormal words whose target similarity with the suspicious segment is greater than a preset similarity threshold, the input text is determined to be normal input.

[0153] For example, assuming that the preset similarity threshold is 0.75, the abnormal words include: abnormal word A, abnormal word B, abnormal word C, the target similarity between abnormal word A and the suspicious segment is 0.4, the target similarity between abnormal word B and the suspicious segment is 0.5, and the target similarity between abnormal word C and the suspicious segment is 0.5. Among the abnormal words, there is no abnormal word whose target similarity with the suspicious segment is greater than the preset similarity threshold of 0.75, then the input text is determined to be normal input.

[0154] Furthermore, in an embodiment of the present application, after obtaining an abnormality detection result of the input text, when the abnormality detection result indicates that the input text is normal input, the input text is input into an image generation model to obtain a target image corresponding to the input text.

[0155] Based on the above embodiments, see Figure 6 FIG. 1 is another flow chart of abnormality detection in an embodiment of the present application, which specifically includes:

[0156] Step 60: Get input text.

[0157] Step 61: Based on the text features of the input text, determine the classification result of the input text.

[0158] Step 62: Determine whether the classification result indicates that the input text is suspicious input. If so, execute step 63; otherwise, execute step 66.

[0159] Step 63: extract suspicious segments from the input text, and calculate the target similarity between the suspicious segments and each abnormal word included in the blacklist.

[0160] Step 64: Determine whether there is an abnormal word among the abnormal words whose target similarity with the suspicious segment is greater than a preset similarity threshold. If so, execute step 65; otherwise, execute step 66.

[0161] Step 65: Determine that the input text is abnormal input, and return abnormal prompt information.

[0162] Step 66: Determine that the input text is normal input, and input the input text into the image generation model to obtain a target image corresponding to the input text.

[0163] Based on the same inventive concept, an abnormality detection device is also provided in the embodiment of the present application, see Figure 7 FIG. 1 is a schematic diagram of the structure of an abnormality detection device in an embodiment of the present application, which specifically includes:

[0164] The acquisition module 701 is used to acquire the input text to be detected;

[0165] A first detection module 702 is used to determine a classification result of the input text based on the text features of the input text, and when the classification result indicates that the input text is suspicious input, extract suspicious segments in the input text;

[0166] The second detection module 703 is used to determine anomaly detection results of the input text based on the target similarity between the suspicious segment and each abnormal word included in the blacklist.

[0167] In a possible embodiment, based on the text features of the input text, a classification result of the input text is determined, and when the classification result indicates that the input text is suspicious input, when extracting suspicious segments in the input text, the first detection module 702 is further configured to:

[0168] Extracting initial text features of input text;

[0169] Based on the attention weight corresponding to the initial text feature, the attention score of each initial text feature element contained in the initial text feature is obtained;

[0170] Based on the obtained attention scores, the initial text features are updated to obtain the target text features, and based on the target text features, the classification results of the input text are obtained;

[0171] When the classification result characterizes the input text as suspicious input, a suspicious segment in the input text is determined based on each attention score.

[0172] In a possible embodiment, the classification result and the suspicious segment are obtained by inputting the input text into the target classification model. The apparatus further includes a training module 704, which is used to:

[0173] Based on the sample input text set, the classification model to be trained is iteratively trained to obtain the target classification model, wherein the sample input text set includes abnormal sample input text and normal sample input text. In one iterative training process, the following operations are performed:

[0174] Extracting sample initial text features of the selected sample input text;

[0175] Based on the sample attention weight corresponding to the sample initial text feature, obtain the sample attention score of each sample initial text feature element contained in the sample initial text feature;

[0176] Based on the attention scores of each sample obtained, the initial text features of the sample are updated to obtain the sample target text features, and based on the sample target text features, the classification prediction results of the sample input text are obtained, and the parameters are adjusted based on the loss values ​​corresponding to the classification prediction results.

[0177] In a possible embodiment, when determining the abnormality detection result of the input text based on the target similarity between the suspicious segment and each abnormal word included in the blacklist, the second detection module 703 is further used to:

[0178] Vectorize the suspicious fragment to obtain the vector to be detected;

[0179] Calculate the cosine similarity between the abnormal vectors of each abnormal word included in the blacklist and the vector to be detected, and obtain the target similarity between each abnormal word and the suspicious segment;

[0180] Based on the obtained similarities of each target, an anomaly detection result of the input text is determined.

[0181] In a possible embodiment, when determining the abnormality detection result of the input text based on the obtained similarities of each target, the second detection module 703 is further used to:

[0182] If there is an abnormal word among the abnormal words whose target similarity with the suspicious segment is greater than a preset similarity threshold, the input text is determined to be an abnormal input;

[0183] If there is no abnormal word among the abnormal words whose target similarity with the suspicious segment is greater than the similarity threshold, the input text is determined to be normal input.

[0184] In a possible embodiment, the second detection module 703 is further configured to:

[0185] When the classification result indicates that the input text is normal input, the input text is input into the image generation model to obtain a target image corresponding to the input text.

[0186] In a possible embodiment, after determining the abnormality detection result of the input text based on the target similarity between the suspicious segment and each abnormal word included in the blacklist, the second detection module 703 is further used to:

[0187] When the anomaly detection result indicates that the input text is normal input, the input text is input into the image generation model to obtain a target image corresponding to the input text;

[0188] When the anomaly detection result indicates that the input text is an abnormal input, an abnormal prompt information is returned, and the abnormal prompt information indicates that the input text is an abnormal input.

[0189] Based on the above embodiments, see Figure 8 Shown is a schematic diagram of the structure of an electronic device in an embodiment of the present application.

[0190] An embodiment of the present application provides an electronic device, which may include a processor 810 (Center Processing Unit, CPU), a memory 820, an input device 830 and an output device 840, etc. The input device 830 may include a keyboard, a mouse, a touch screen, etc., and the output device 840 may include a display device, such as a liquid crystal display (Liquid Crystal Display, LCD), a cathode ray tube (Cathode Ray Tube, CRT), etc.

[0191] The memory 820 may include a read-only memory (ROM) and a random access memory (RAM), and provides the processor 810 with program instructions and data stored in the memory 820. In the embodiment of the present application, the memory 820 may be used to store the program of any abnormality detection method in the embodiment of the present application.

[0192] The processor 810 calls the program instructions stored in the memory 820, and the processor 810 is used to execute any one of the abnormality detection methods in the embodiments of the present application according to the obtained program instructions.

[0193] Based on the above embodiments, in an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the anomaly detection method in any of the above method embodiments is implemented.

[0194] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0195] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0196] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0197] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0198] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. An anomaly detection method, It is characterized in that include: Get the input text to be detected; Determining a classification result of the input text based on text features of the input text, and extracting suspicious segments from the input text when the classification result indicates that the input text is suspicious input; Based on the target similarity between the suspicious segment and each abnormal word included in the blacklist, an abnormality detection result of the input text is determined.

2. The method according to claim 1, It is characterized in that The step of determining a classification result of the input text based on the text feature of the input text, and extracting a suspicious segment from the input text when the classification result indicates that the input text is suspicious input, includes: Extracting initial text features of the input text; Based on the attention weight corresponding to the initial text feature, obtaining the attention score of each initial text feature element contained in the initial text feature; Based on the obtained attention scores, the initial text features are updated to obtain target text features, and based on the target text features, a classification result of the input text is obtained; When the classification result indicates that the input text is suspicious input, a suspicious segment in the input text is determined based on the attention scores.

3. The method according to claim 1 or 2, It is characterized in that The classification result and the suspicious segment are obtained by inputting the input text into a target classification model, wherein the target classification model is trained in the following manner: Based on the sample input text set, the classification model to be trained is iteratively trained to obtain a target classification model, wherein the sample input text set includes abnormal sample input text and normal sample input text, and in one iterative training process, the following operations are performed: Extracting sample initial text features of the selected sample input text; Based on the sample attention weight corresponding to the sample initial text feature, obtaining the sample attention score of each sample initial text feature element included in the sample initial text feature; Based on the attention scores of each sample obtained, the initial text features of the sample are updated to obtain the sample target text features, and based on the sample target text features, the classification prediction results of the sample input text are obtained, and the parameters are adjusted based on the loss values ​​corresponding to the classification prediction results.

4. The method according to claim 1, It is characterized in that The determining of the abnormality detection result of the input text based on the target similarity between the suspicious segment and each abnormal word included in the blacklist includes: Vectorizing the suspicious segment to obtain a vector to be detected; Calculating the cosine similarity between each abnormal vector of each abnormal word included in the blacklist and the vector to be detected, and obtaining the target similarity between each abnormal word and the suspicious segment; Based on the obtained similarities of each target, an abnormality detection result of the input text is determined.

5. The method according to claim 4, It is characterized in that The step of determining the abnormality detection result of the input text based on the obtained target similarities includes: If there is an abnormal word among the abnormal words whose target similarity with the suspicious segment is greater than a preset similarity threshold, then determining that the input text is an abnormal input; If there is no abnormal word among the abnormal words whose target similarity with the suspicious segment is greater than the similarity threshold, it is determined that the input text is normal input.

6. The method according to claim 1, It is characterized in that The method further comprises: When the classification result indicates that the input text is normal input, the input text is input into an image generation model to obtain a target image corresponding to the input text.

7. The method according to claim 1, It is characterized in that After determining the abnormality detection result of the input text based on the target similarity between the suspicious segment and each abnormal word included in the blacklist, the method further includes: When the abnormality detection result indicates that the input text is normal input, inputting the input text into an image generation model to obtain a target image corresponding to the input text; When the abnormality detection result indicates that the input text is abnormal input, abnormality prompt information is returned, where the abnormality prompt information indicates that the input text is abnormal input.

8. An abnormality detection device, It is characterized in that include: An acquisition module is used to acquire input text to be detected; A first detection module, configured to determine a classification result of the input text based on text features of the input text, and extract suspicious segments from the input text when the classification result indicates that the input text is suspicious input; The second detection module is used to determine the abnormality detection result of the input text based on the target similarity between the suspicious segment and each abnormal word included in the blacklist.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, Features: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images