Text processing method and device, equipment, storage medium and program product

Abstract

The invention provides a text processing method and device, equipment, a storage medium and a program product, which are applied to various text-based information generation scenes such as cloud technology, artificial intelligence, intelligent transportation and information recommendation. The text processing method comprises the steps that in response to an information generation request, a to-be-detected text is obtained, the to-be-detected text is a prompt text, and the to-be-detected text is used for generating information of a target type; encoding the to-be-detected text into to-be-detected features; on the basis of the to-be-detected features, N anomaly probabilities corresponding to N anomaly categories are predicted, N is a positive integer, and the anomaly categories refer to problem categories existing in the information of the target type; when M abnormal probabilities in the N abnormal probabilities are smaller than respective corresponding probability threshold values, first target information is generated based on the to-be-detected text, M is a positive integer, and the first target information is information of a target type. According to the invention, the information generation efficiency can be improved.

Description

Technical Field

[0001] The present application relates to natural language processing technology in the field of artificial intelligence, and in particular to a text processing method, device, equipment, storage medium and program product. Background Art

[0002] In the process of artificial intelligence application, it is often necessary to generate target type information based on prompt text and perform subsequent processing on the generated information. However, the generated information often has quality problems and needs to be regenerated, which affects the efficiency of information generation. Summary of the invention

[0003] The embodiments of the present application provide a text processing method, apparatus, device, computer-readable storage medium, and computer program product, which can improve the efficiency of information generation.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] The present application provides a text processing method, which includes:

[0006] In response to the information generation request, obtaining a text to be detected, wherein the text to be detected is a prompt text, and the text to be detected is used to generate information of a target type;

[0007] Encoding the text to be detected as features to be detected;

[0008] Based on the features to be detected, predict N abnormal probabilities corresponding to N abnormal categories, where N is a positive integer, and the abnormal category refers to the problem category existing in the information of the target type;

[0009] When M of the abnormal probabilities among the N abnormal probabilities are less than their respective corresponding probability thresholds, first target information is generated based on the text to be detected, where M≤N and M is a positive integer, and the first target information is information of the target type.

[0010] The present application provides a text processing device, the text processing device comprising:

[0011] A request response module, used for obtaining a text to be detected in response to an information generation request, wherein the text to be detected is a prompt text, and the text to be detected is used to generate information of a target type;

[0012] A feature extraction module, used for encoding the text to be detected into features to be detected;

[0013] An anomaly prediction module, used to predict N anomaly probabilities corresponding to N anomaly categories based on the features to be detected, where N is a positive integer and the anomaly category refers to a problem category existing in the information of the target type;

[0014] An information generation module is used to generate first target information based on the text to be detected when M of the N abnormal probabilities are less than their respective corresponding probability thresholds, wherein M≤N, and M is a positive integer, and the first target information is information of the target type.

[0015] In an embodiment of the present application, the text processing device also includes an information correction module, which is used to correct the text to be detected when L of the abnormal probabilities among the N abnormal probabilities are greater than or equal to the corresponding probability thresholds, so as to obtain a first prompt text, wherein L≤N, and L is a positive integer; and generate second target information based on the first prompt text, wherein the second target information is information of the target type.

[0016] In an embodiment of the present application, the information generation module is also used to determine P abnormal categories corresponding to the L abnormal probabilities when L abnormal probabilities among the N abnormal probabilities are greater than or equal to the corresponding probability thresholds, wherein P≤N and P is a positive integer; generate third target information based on the text to be detected, wherein the third target information is information of the target type; perform P types of abnormality detection corresponding to the P abnormal categories on the third target information; and when it is detected that the third target information does not contain information belonging to any of the P abnormal categories, determine the third target information as the final target information.

[0017] In an embodiment of the present application, the information generation module is also used to determine L abnormal categories corresponding to the L abnormal probabilities when L is less than the minimum number of categories, and determine the L abnormal categories as P abnormal categories; when L is greater than or equal to the minimum number of categories, select the abnormal probability of the minimum number of categories with the largest abnormal probability from the L abnormal probabilities, and determine the abnormal category of the minimum number of categories corresponding to the abnormal probability of the minimum number of categories as P abnormal categories.

[0018] In an embodiment of the present application, the information correction module is also used to correct the text to be detected to obtain a second prompt text when it is detected that the third target information contains information belonging to any of the P abnormal categories; and generate fourth target information based on the second prompt text, wherein the fourth target information is information of the target type.

[0019] In an embodiment of the present application, the feature extraction module is also used to preprocess the text to be detected to obtain the text to be encoded; segment the text to be encoded to obtain a sequence of segmented words to be encoded; encode each segmented word to be encoded in the sequence of segmented words to be encoded to obtain a segmented word encoding sequence; and perform vector conversion on the segmented word encoding sequence to obtain the feature to be detected.

[0020] In an embodiment of the present application, the N abnormal probabilities are obtained through a text detection model, and the text processing device also includes a model training module for obtaining training data, wherein the training data includes sample text and abnormal category labels; obtaining the abnormal category prediction result corresponding to the sample text based on the model to be trained, wherein the model to be trained is an artificial neural network model to be trained for detecting prompt text; determining a loss function value based on the difference between the abnormal category prediction result and the abnormal category label; and training the model to be trained based on the loss function value to obtain the text detection model.

[0021] In an embodiment of the present application, the model training module is also used to generate information for the sample text to obtain target sample information; perform anomaly detection on the target sample information to obtain the anomaly category label; and combine the sample text and the anomaly category label into the training data.

[0022] In an embodiment of the present application, the model training module is also used to perform the following processing for the predicted probability of each abnormal category in the abnormal category prediction result: determine the abnormal label corresponding to the predicted probability from the abnormal category label; calculate the category loss function value based on the difference between the predicted probability and the abnormal label; obtain multiple category loss function values ​​corresponding to N abnormal categories from the category loss function value of each abnormal category; and integrate multiple category loss function values ​​into the loss function value.

[0023] In an embodiment of the present application, the target type includes at least one of the following: image type, text type, voice type, and video type.

[0024] In an embodiment of the present application, when the target type is an image type, the N abnormality categories include at least one of the following: abnormal generation method, abnormal clarity, abnormal morphology, and abnormal scene type, wherein the abnormal generation method means that the generated image is non-native, the abnormal clarity means that the clarity of the generated image is lower than the clarity threshold, the abnormal morphology means that the distortion of the generated image is higher than the distortion threshold, and the abnormal scene type means that the boundary value between the foreground and background of the generated image is lower than the boundary threshold.

[0025] An embodiment of the present application provides an electronic device for text processing, the electronic device comprising:

[0026] A memory for storing computer executable instructions or computer programs;

[0027] The processor is used to implement the text processing method provided in the embodiment of the present application when executing the computer executable instructions or computer programs stored in the memory.

[0028] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions or a computer program, wherein the computer-executable instructions or the computer program are used to implement the text processing method provided in the embodiment of the present application when executed by a processor.

[0029] An embodiment of the present application provides a computer program product, including computer executable instructions or a computer program. When the computer executable instructions or the computer program are executed by a processor, the text processing method provided by the embodiment of the present application is implemented.

[0030] The embodiments of the present application have at least the following beneficial effects: before generating information of the target type based on the text to be detected, the text to be detected is first detected to obtain N abnormality probabilities corresponding to the N abnormal categories; since the target type information is generated when the M abnormality probabilities are less than their respective corresponding probability thresholds, it is ensured that the information generation is performed after the text to be detected passes the detection, and the information generation can be performed accurately, and the quality of the information generation is improved, thereby improving the efficiency of the information generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a schematic diagram of the architecture of the text processing system provided in the embodiment of the present application;

[0032] Figure 2 This is a method provided by the embodiment of the present application. Figure 1 A schematic diagram of the structure of the server in FIG.

[0033] Figure 3 This is a flowchart of the text processing method provided in the embodiment of the present application. Figure 1 ;

[0034] Figure 4 This is a flowchart of the text processing method provided in the embodiment of the present application. Figure 2 ;

[0035] Figure 5 This is a flowchart of the text processing method provided in the embodiment of the present application. Figure 3 ;

[0036] Figure 6is a schematic diagram of an exemplary model training process provided in an embodiment of the present application;

[0037] Figure 7 is a schematic diagram of an exemplary Wensheng diagram process provided in an embodiment of the present application;

[0038] Figure 8 is an exemplary prompt text quality detection model training diagram provided in an embodiment of the present application;

[0039] Fig. 9 is an exemplary label schematic diagram provided in an embodiment of the present application;

[0040] Fig.10 It is an exemplary quality detection model provided in the embodiments of the present application;

[0041] Fig.11 It is a schematic diagram of an exemplary detection indicator trend provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0043] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0044] In the following description, the terms "first\second\third\fourth" are used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third\fourth" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0045] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0046] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0047] 1) Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. The text processing method provided in the embodiment of the present application is applied to the field of artificial intelligence.

[0048] 2) Machine Learning (ML) is a multi-disciplinary interdisciplinary subject involving probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It is used to study computer simulation or realization of human learning behavior to acquire new knowledge or skills; reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Machine learning applications are spread across all areas of artificial intelligence. Machine learning generally includes technologies such as artificial neural networks, belief networks, reinforcement learning, transfer learning, and inductive learning. The text processing method provided in the embodiment of the present application can be implemented by an artificial neural network in machine learning.

[0049] 3) Artificial neural network is a mathematical model that imitates the structure and function of biological neural networks. The exemplary structures of artificial neural networks in the embodiments of the present application include graph convolutional networks (Graph Convolutional Network, GCN, a neural network for processing graph-structured data), deep neural networks (Deep Neural Networks, DNN), convolutional neural networks (Convolutional Neural Network, CNN) and recurrent neural networks (Recurrent Neural Network, RNN), neural state machines (Neural State Machine, NSM) and phase function neural networks (Phase-Functioned Neural Network, PFNN), etc. The models to be trained, text detection models and information generation models involved in the embodiments of the present application are all artificial neural network models.

[0050] 4) Text-generated images refer to the process of generating images based on text. Text-generated images can be generated through artificial neural network models. In this case, text-generated images belong to generative artificial intelligence (AIGC).

[0051] 5) A "Prompt" refers to the input text of an artificial neural network model, which can be an instruction description text or a question description text, and is used to control the processing method of the artificial neural network model to obtain a specific output result. Among them, the "text" in text-to-image, the "text" in text-to-text, the "text" in text-to-audio, and the "text" in text-to-video are all prompts.

[0052] It should be noted that in the process of artificial intelligence applications, it is often necessary to generate information of a target type based on a prompt and perform subsequent processing on the generated information. Among them, when performing subsequent processing, first evaluate the quality of the generated information. For example, evaluate the quality of the generated image from dimensions such as whether it is original, clear, and distorted; then, filter out low-quality information from the generated information and apply the remaining high-quality information to downstream tasks. However, the generated information often has quality problems, resulting in the number of high-quality information obtained being lower than the specified number and needing to be regenerated. Therefore, the efficiency of information generation is affected.

[0053] Based on this, the embodiments of the present application provide a text processing method, device, equipment, computer-readable storage medium, and computer program product, which can improve the efficiency of information generation. The following describes the exemplary application of the equipment provided by the embodiments of the present application. The electronic device for text processing provided by the embodiments of the present application (hereinafter referred to as the text processing device) can be implemented as various types of terminals such as smartphones, smart watches, laptop computers, tablet computers, desktop computers, smart home appliances, set-top boxes, intelligent vehicle-mounted devices, portable music players, personal digital assistants, dedicated messaging devices, intelligent voice interaction devices, portable game devices, and smart speakers, or can be implemented as a server, or can be a combination of the two. The following will describe the exemplary application when the text processing device is implemented as a server.

[0054] See Figure 1 , Figure 1 is the schematic diagram of the architecture of the text processing system provided by the embodiments of the present application; as Figure 1 shown, to support a text processing application, in the text processing system 100, terminals 200 (exemplarily showing terminals 200-1 and 200-2) are connected to the server 400 through the network 300. The network 300 can be a wide area network, a local area network, or a combination of the two. In addition, the text processing system 100 also includes a database 500 for providing data support to the server 400; and, Figure 1 shown in is a case where the database 500 is independent of the server 400. In addition, the database 500 can also be integrated in the server 400, and the embodiments of the present application do not limit this.

[0055] The server 400 is used to receive an information generation request sent by the terminal 200 through the network 300, and obtain a text to be detected in response to the information generation request, where the text to be detected is a prompt text and is used to generate information of the target type; encode the text to be detected into features to be detected; based on the features to be detected, predict N abnormal probabilities corresponding to N abnormal categories, where N is a positive integer, and the abnormal category refers to the problem category of the information of the target type; when M abnormal probabilities among the N abnormal probabilities are less than their respective corresponding probability thresholds, generate first target information based on the text to be detected, where M≤N, and M is a positive integer, and the first target information is information of the target type; and send the first target information to the terminal 200 through the network 300.

[0056] Terminal 200 is used to generate a request based on the received operation generation information and send an information request to server 400 through network 300; it is also used to receive the first target information sent by server 400 through network 300 and display the first target information (graphic interface 210-1 and graphical interface 210-2 are shown as examples).

[0057] In some embodiments, the server 400 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0058] See also Figure 2 , Figure 2 This is a method provided by the embodiment of the present application. Figure 1 The structural diagram of the server in Figure 2 As shown, the server 400 includes: at least one processor 410, a memory 450 and at least one network interface 420. The various components in the terminal 400 are coupled together through a bus system 440. It can be understood that the bus system 440 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Figure 2 Various buses are labeled as bus system 440 .

[0059] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0060] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0061] The memory 450 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0062] In some embodiments, memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0063] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0064] A network communication module 452, for reaching other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 include: Bluetooth, Wireless Fidelity (Wi-Fi), and Universal Serial Bus (USB), etc.;

[0065] In some embodiments, the text processing device provided in the embodiments of the present application can be implemented in software. Figure 2 The text processing device 455 stored in the memory 450 is shown, which can be software in the form of a program and a plug-in, etc., including the following software modules: a request response module 4551, a feature extraction module 4552, an abnormal prediction module 4553, an information generation module 4554, an information correction module 4555 and a model training module 4556. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be explained below.

[0066] In some embodiments, the text processing device provided in the embodiments of the present application can be implemented in hardware. As an example, the text processing device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the text processing method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application specific integrated circuits (Application Specific Integrated Circuit, ASIC), DSP, programmable logic device (Programmable Logic Device, PLD), complex programmable logic device (Complex Programmable Logic Device, CPLD), field programmable gate array (Field-Programmable Gate Array, FPGA) or other electronic components.

[0067] In some embodiments, the terminal or server can implement the text processing method provided in the embodiment of the present application by running various computer executable instructions or computer programs. For example, computer executable instructions can be commands, machine instructions or software instructions at the microprogram level. The computer program can be a native program or software module in the operating system; it can be a native (Native) application (APPlication, APP), that is, a program that needs to be installed in the operating system to run, such as Vincent Figure APP, or Vincent Text APP, or Vincent Video APP, etc.; it can also be a small program that can be embedded in any APP, that is, a program that can be run only by downloading it to a browser environment. In short, the above-mentioned computer executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, module or plug-in in any form.

[0068] Below, the text processing method provided by the embodiment of the present application will be described in combination with the exemplary application and implementation of the text processing device provided by the embodiment of the present application. In addition, the text processing method provided by the embodiment of the present application is applied to various text-based information generation scenarios such as cloud technology, artificial intelligence, smart transportation, and information recommendation.

[0069] See also Figure 3 , Figure 3 This is a flowchart of the text processing method provided in the embodiment of the present application. Figure 1 ,in, Figure 3 The execution subject of each step in the text processing device is the text processing device; Figure 3 The steps shown are explained.

[0070] Step 101: In response to an information generation request, obtain a text to be detected.

[0071] In the embodiment of the present application, when the text processing device receives a request to generate information based on the prompt text, it also receives the information generation request; at this time, the text processing device responds to the information generation request and starts to execute the processing flow of information generation. Since the information generation request is a request to generate information based on the prompt text, the text processing device can obtain the prompt text from the information generation request, wherein the obtained prompt text is the text to be detected.

[0072] It should be noted that the text to be detected is a prompt text, and the text to be detected is used to generate information of the target type; wherein the target type includes at least one of the following: image type, text type, voice type, and video type. When the target type is an image type, the text processing method is applied to a text-image scenario, for example, the advertiser's needs are described as a prompt text, and then an advertising image that meets the advertiser's needs is generated based on the prompt text; when the target type is a text type, the text processing method is applied to a text-text scenario, for example, the requirements of the text to be generated are described as a prompt text, and then a target text (such as an article, etc.) that meets the requirements is generated based on the prompt text; when the target type is a voice type, the text processing method is applied to a text-voice scenario, for example, the requirements of the voice to be generated are described as a prompt text, and then a target voice that meets the requirements is generated based on the prompt text; when the target type is a video type, the text processing method is applied to a text-video scenario, for example, the requirements of the video to be generated are described as a prompt text, and then a target video that meets the requirements is generated based on the prompt text.

[0073] Step 102: Encode the text to be detected into features to be detected.

[0074] It should be noted that the text processing device extracts the features of the text to be detected by encoding the text to be detected, thereby obtaining the features to be detected. Since the text to be detected is a prompt text to be quality-detected from the dimension of the generated information, the features to be detected are used to perform quality detection on the text to be detected from the dimension of the generated information.

[0075] In step 102 of the embodiment of the present application, the text processing device encodes the text to be detected into the feature to be detected, including: the text processing device first preprocesses the text to be detected to obtain the text to be encoded; then segments the text to be encoded to obtain a sequence of segmented words to be encoded; then, each segmented word to be encoded in the sequence of segmented words to be encoded is encoded to obtain a segmented word encoding sequence; finally, the segmented word encoding sequence is vectorized to obtain the feature to be detected.

[0076] It should be noted that the preprocessing includes at least one of the following: removing punctuation marks, unifying upper and lower cases, removing auxiliary words, etc.; the preprocessing method may vary depending on the application scenario. For example, when the text to be detected is in Chinese, the preprocessing may be to remove punctuation marks in the text to be detected; when the text to be detected is in English, the preprocessing may be to unify upper and lower cases; and so on. In the sequence of word segmentation to be encoded, each element is a word segmentation to be encoded, and the word segmentation to be encoded is a word or a string; in the word segmentation code sequence, each element is a word segmentation code, and the word segmentation code is the initial encoding result of the word segmentation to be encoded. Here, vector transformation is used to extract the features of the word segmentation code sequence.

[0077] Step 103: Based on the features to be detected, predict N abnormality probabilities corresponding to the N abnormality categories.

[0078] In an embodiment of the present application, the text processing device predetermines N abnormal categories for information of the target type, so that the text processing device performs quality detection on the text to be detected based on the features to be detected, so as to predict the probability of the existence of each abnormal category in the text to be detected based on the features to be detected, and the predicted probability is called the abnormal probability corresponding to the abnormal category; thus, the text processing device can predict N abnormal probabilities corresponding to N abnormal categories; wherein N is a positive integer, and the N abnormal categories correspond one to one to the N abnormal probabilities. Here, the text processing device can directly predict the N abnormal probabilities corresponding to the N abnormal categories based on the features to be detected, and can also obtain the N abnormal probabilities of the text to be detected through an artificial neural network model, which is not limited in the embodiment of the present application.

[0079] It should be noted that the abnormality category refers to the problem category of the target type information; when the target type is an image type, the N abnormality categories include at least one of the following: abnormal generation method, abnormal clarity, abnormal morphology, and abnormal scene, where the abnormal generation method means that the generated image is not native, the abnormal clarity means that the clarity of the generated image is lower than the clarity threshold, the abnormal morphology means that the distortion of the generated image is higher than the distortion threshold, and the abnormal scene means that the boundary value between the foreground and background of the generated image is lower than the boundary threshold. Here, the foreground refers to the main body of the image, the background refers to the background of the image, and the boundary value between the foreground and background refers to the difference in clarity between the main body of the image and the background of the image.

[0080] Step 104: When M abnormal probabilities among the N abnormal probabilities are less than their respective corresponding probability thresholds, first target information is generated based on the text to be detected.

[0081] In the embodiment of the present application, each abnormal type corresponds to a probability threshold, and N abnormal types correspond to N probability thresholds. The N probability thresholds may be the same or different, and the embodiment of the present application does not limit this. Here, the text processing device can determine that the quality inspection of the text to be detected has passed when M abnormal probabilities among the N abnormal probabilities are less than their respective corresponding probability thresholds. At this time, the target type information is generated based on the text to be detected, and the generated result is the first target information; thus, when the M abnormal probabilities are less than their respective corresponding probability thresholds, the first target information corresponds to the text to be detected; here, the text processing device determines the first target information as the final target information, and the final target information is the response result of the information generation request.

[0082] It should be noted that M≤N, and M is a positive integer, so that the M abnormal probabilities can be all of the N abnormal probabilities, or part of the N abnormal probabilities, and the relationship between M and N can be determined according to the actual implementation situation; wherein, when the M abnormal probabilities are part of the N abnormal probabilities, the M abnormal probabilities are less than their respective corresponding probability thresholds, indicating that the NM abnormal probabilities are greater than or equal to their respective corresponding probability thresholds; here, M can also be greater than the first threshold, which represents the maximum number of abnormal categories corresponding to the situation where the quality detection passes, is positively correlated with L, and can be determined according to the actual implementation situation. In addition, the abnormal probability is positively correlated with the degree of abnormality of the abnormal category. The first target information is information of the target type, such as images, text, voice, video, etc.

[0083] It can be understood that before generating information of the target type based on the text to be detected, the text to be detected is first detected to obtain N abnormal probabilities corresponding to the N abnormal categories; since the target type information is generated when the M abnormal probabilities are less than their respective corresponding probability thresholds, that is, information generation is performed only when it is determined that the text to be detected has passed the quality test, information generation can be performed accurately, the quality of information generation can be improved, and thus, the efficiency of information generation can be improved.

[0084] See also Figure 4 , Figure 4 This is a flowchart of the text processing method provided in the embodiment of the present application. Figure 2 ,in, Figure 4 The execution subject of each step in is the text processing device; Figure 4 As shown, in the embodiment of the present application, step 103 also includes step 105 and step 106; that is, after the text processing device predicts N abnormal probabilities corresponding to N abnormal categories based on the features to be detected, the text processing method also includes step 105 and step 106, and each step is explained below.

[0085] Step 105: When L abnormal probabilities among the N abnormal probabilities are greater than or equal to their corresponding probability thresholds, the text to be detected is corrected to obtain a first prompt text.

[0086] In an embodiment of the present application, the text processing device can determine that the quality inspection of the text to be detected has not passed when L abnormal probabilities among N abnormal probabilities are greater than or equal to their respective corresponding probability thresholds. At this time, the information generation process of the text to be detected is terminated, and the text to be detected is corrected. The corrected text to be detected obtained is the first prompt text. Here, the text processing device makes targeted corrections to the text to be detected based on the L abnormal categories corresponding to the L abnormal probabilities. The targeted correction can be achieved through a trained natural language processing model, by refining the text to be detected, or by adding L abnormal categories to the negative prompt text (Negative Prompt) of the text to be detected, and so on. The embodiment of the present application does not limit this; wherein the negative prompt text is used to control the generated information to deviate from the descriptive semantics of the negative prompt text).

[0087] It is understandable that, through targeted correction, the correction efficiency of the text to be detected can be improved, thereby improving the quality of the first prompt text.

[0088] It should be noted that the first prompt text is a prompt text, and the first prompt text is used to generate information of the target type. L≤N, and L is a positive integer, so that the L abnormal probabilities can be all of the N abnormal probabilities, or part of the N abnormal probabilities, and the relationship between L and N can be determined according to the actual implementation situation. In addition, when the L abnormal probabilities are part of the N abnormal probabilities, the L abnormal probabilities are greater than or equal to their respective corresponding probability thresholds, indicating that the NL abnormal probabilities are less than their respective corresponding probability thresholds; here, L can also be greater than the second threshold, and the second threshold is positively correlated with L, indicating that the maximum number of abnormal categories corresponding to the situation where the quality inspection passes can be determined according to the actual implementation situation; and, L and M can be the same or different.

[0089] Step 106: Generate second target information based on the first prompt text.

[0090] In the embodiment of the present application, the text processing device generates information of the target type using the first prompt text, and the generated information is the second target information; thus, when the L abnormal probabilities are greater than or equal to the probability thresholds corresponding to each other, the second target information corresponds to the text to be detected. Here, the text processing device determines the second target information as the final target information.

[0091] It should be noted that after obtaining the first prompt text, the text processing device can directly generate information of the target type based on the first prompt text, and can also determine that the quality test of the first prompt text has passed before generating information of the target type; wherein, the process of quality testing the first prompt text is similar to the process of quality testing the text to be detected, and the process of determining that the quality test of the first prompt text has passed is similar to the process of determining that the quality test of the text to be detected has passed, and the embodiments of the present application will not be repeated here. In addition, when the text processing device determines that the quality test of the first prompt text has not passed, the first prompt text can continue to be corrected; since the process of correcting the first prompt text is similar to the process of correcting the text to be detected, the embodiments of the present application will not be repeated here. wherein, the second target information is information of the target type, such as images, text, voice, video, etc.

[0092] It can be understood that when it is determined that L abnormal probabilities among N abnormal probabilities are greater than or equal to their respective corresponding probability thresholds, it is determined that the quality test of the text to be detected has failed; at this time, targeted corrections are made to the text to be detected based on the L abnormal types corresponding to the L abnormal probabilities, which can improve the quality of the text to be detected, and then when information is generated based on the corrected second target information, the quality and efficiency of information generation can be improved.

[0093] See also Figure 5 , Figure 5 This is a flowchart of the text processing method provided in the embodiment of the present application. Figure 3 ,in, Figure 5 The execution subject of each step in is the text processing device; Figure 5 As shown, in the embodiment of the present application, step 103 also includes steps 107 to 110; that is, after the text processing device predicts N abnormal probabilities corresponding to N abnormal categories based on the features to be detected, the text processing method also includes steps 107 to 110, and each step is explained below.

[0094] Step 107: When L abnormal probabilities among the N abnormal probabilities are greater than or equal to their corresponding probability thresholds, P abnormal categories corresponding to the L abnormal probabilities are determined.

[0095] In the embodiment of the present application, when the text processing device determines that L of the N abnormal probabilities are greater than or equal to the probability thresholds corresponding to each other, the text to be detected can be directly corrected, and the information generated by the text to be detected can also be targeted for detection, which is not limited in the embodiment of the present application. When the text processing device fails the quality detection of the text to be detected and performs targeted detection on the information generated by the text to be detected, P abnormal categories for targeted detection are first determined; where P≤N, and P is a positive integer.

[0096] It should be noted that the text processing device can directly determine the L abnormal categories corresponding to the L abnormal probabilities as P abnormal categories, and can also select P abnormal categories from the L abnormal categories based on the minimum number of categories, and so on. The embodiments of the present application are not limited to this.

[0097] In an embodiment of the present application, a text processing device determines P abnormality categories corresponding to L abnormality probabilities, including: when L is less than a minimum number of categories, the text processing device determines L abnormality categories corresponding to the L abnormality probabilities, and determines the L abnormality categories as P abnormality categories; or, when L is greater than or equal to the minimum number of categories, selecting an abnormality probability of the lowest number of categories with the largest abnormality probability from the L abnormality probabilities, and determining the abnormality category of the lowest number of categories corresponding to the abnormality probability of the lowest number of categories as P abnormality categories.

[0098] It should be noted that the text processing device can obtain the minimum number of categories, which represents the minimum number of abnormal categories for accurately determining the quality of the text to be detected, such as 5. The minimum number of categories can be set according to actual conditions; thus, the text processing device compares L with the number of abnormal categories to determine the P abnormal categories to be detected based on the comparison result. Here, when L is less than the minimum number of categories, it indicates that the probability that the text to be detected is a low-quality prompt text is low, so at this time, the text processing device performs L abnormal category detection on the text to be detected; and when L is greater than or equal to the minimum number of categories, it indicates that the probability that the text to be detected is a low-quality prompt text is high, and because the abnormal category detection using the maximum number of categories can determine the accuracy of the abnormal category detection, at this time, the text processing device selects P abnormal categories from the L abnormal categories based on the abnormal probability to achieve targeted detection of P abnormal categories.

[0099] It can be understood that by comparing L minimum number of categories, when L is greater than or equal to the minimum number of categories, P abnormal categories are selected therefrom to perform P types of abnormality detection on the third target information; since the minimum number of categories is the minimum number of abnormal categories to ensure the quality of abnormality detection, the number of abnormality detections can be reduced while ensuring detection accuracy, thereby improving detection efficiency.

[0100] Step 108: Generate third target information based on the text to be detected.

[0101] It should be noted that in order to perform targeted detection on the generated information of the text to be detected, the text processing device generates target type information based on the text to be detected, and the generated information is the third target information; wherein, the third target information is information of the target type, such as image, text, voice, video, etc.; it is easy to know that when L abnormal probabilities are greater than or equal to their respective corresponding probability thresholds, the third target information corresponds to the text to be detected.

[0102] Step 109: Perform P types of anomaly detection corresponding to the P anomaly categories on the third target information.

[0103] It should be noted that each anomaly category corresponds to a corresponding anomaly detection, so N anomaly types correspond to N anomaly detections; thus, P anomaly categories correspond to P anomaly detections; and further, the text processing device performs P anomaly detections on the third target information.

[0104] Step 110: When it is detected that the third target information does not contain any information belonging to any of the P abnormal categories, the third target information is determined as the final target information.

[0105] It should be noted that when the text processing device completes P types of abnormality detection on the third target information, and detects that the third target information does not contain information belonging to any of the P abnormal categories, it indicates that the quality detection of the third target information has passed, and thus, the text processing device can determine the third target information as the final target information.

[0106] Continue to see Figure 5 In the embodiment of the present application, step 109 also includes step 111 and step 112; that is, after the text processing device performs P types of anomaly detection corresponding to P anomaly categories on the third target information, the text processing method also includes step 111 and step 112, and each step is explained below.

[0107] Step 111: When it is detected that the third target information contains information belonging to any abnormal category among the P abnormal categories, the text to be detected is corrected to obtain a second prompt text.

[0108] It should be noted that when the text processing device completes P types of abnormality detection on the third target information, and detects that the third target information contains information belonging to any of the P abnormal categories, it is determined that the third target information has a quality problem, and it is caused by the quality problem of the text to be detected; thus, at this time, the text processing device performs targeted corrections on the text to be detected based on the abnormal category of the third target information, and the correction result obtained is the second prompt text. The second prompt text is a prompt text used to generate information of the target type.

[0109] Step 112: Generate fourth target information based on the second prompt text.

[0110] In an embodiment of the present application, since the second prompt text is obtained by correcting the text to be detected, the quality of the second prompt text can be ensured; then, the text processing device generates target type information based on the second prompt text, and the generated information is the fourth target information; wherein the fourth target information is target type information, such as image, text, voice, video, etc.

[0111] It should be noted that after obtaining the second prompt text, the text processing device can directly generate information of the target type based on the second prompt text, and can also determine that the quality test of the second prompt text has passed before generating information of the target type; wherein, the process of quality testing the second prompt text is similar to the process of quality testing the text to be tested, and the process of determining that the quality test of the second prompt text has passed is similar to the process of determining that the quality test of the text to be tested has passed, and the embodiments of the present application will not be repeated here. In addition, when the text processing device determines that the quality test of the second prompt text has not passed, the second prompt text can continue to be corrected; since the process of correcting the second prompt text is similar to the process of correcting the text to be tested, the embodiments of the present application will not be repeated here.

[0112] In an embodiment of the present application, when the process of obtaining N abnormality probabilities based on the text to be detected is implemented by an artificial neural network model, the artificial neural network model passed is a text detection model; the text detection model is used to predict the probabilities of the prompt text corresponding to N abnormal categories. When a prompt text is given, the prompt text is input into the text detection model, and the probabilities corresponding to N abnormal categories can be output from the text detection model; when the given prompt text is the text to be detected, by inputting the text to be detected into the text detection model, N abnormality probabilities corresponding to N abnormal categories can be output from the text detection model; that is, the text detection model is used to perform the following processing: obtaining the features to be detected of the text to be detected, and predicting N abnormality probabilities corresponding to N abnormal categories based on the text to be detected.

[0113] See also Figure 6 , Figure 6 is a schematic diagram of an exemplary model training process provided in an embodiment of the present application, wherein the execution subject of each step is a text processing device; Figure 6 As shown, the text detection model is implemented through steps 113 to 116, and each step is described below.

[0114] Step 113: Obtain training data, where the training data includes sample text and abnormal category labels.

[0115] It should be noted that training data refers to the data set used for model training; sample text is the prompt text used for model training, and the sample text is used to generate information of the target type; exception category label refers to the N exception labels corresponding to the sample text and N exception categories, and each exception label indicates whether there is an anomaly represented by the corresponding exception category.

[0116] In an embodiment of the present application, a text processing device obtains training data, including: the text processing device first generates information on a sample text to obtain target sample information; then performs anomaly detection on the target sample information to obtain anomaly category labels; and finally combines the sample text and the anomaly category labels into training data.

[0117] It should be noted that the text processing device can directly generate information of the target type on the sample text to obtain the target sample information, and can also generate information based on an artificial neural network model for generating information of the target type, etc., which is not limited in the embodiments of the present application. Here, the text processing device can perform N types of anomaly detection on the target sample information to obtain N anomaly labels corresponding to the N types of anomaly detection, and the N anomaly labels are referred to as anomaly category labels.

[0118] Step 114: Obtain anomaly category prediction results corresponding to the sample text based on the model to be trained.

[0119] In the embodiment of the present application, the text processing device uses the model to be trained to perform quality detection on the sample text to predict N prediction probabilities corresponding to N abnormal categories of the sample text, and determines the N prediction probabilities as the abnormal category prediction results corresponding to the sample text. Each prediction probability represents the predicted probability of the corresponding abnormal category, which is used to determine whether the sample text has an abnormality of the corresponding abnormal category.

[0120] It should be noted that the model to be trained is an artificial neural network model to be trained for detecting prompt text. It can be an original artificial neural network model constructed, or a pre-trained artificial neural network model, etc. The embodiment of the present application does not limit this.

[0121] Step 115: Determine a loss function value based on the difference between the abnormal category prediction result and the abnormal category label.

[0122] It should be noted that after the text processing device obtains the abnormal category prediction result, it compares the abnormal category prediction result with the abnormal category label to obtain the difference between the abnormal category prediction result and the abnormal category label; since the difference between the abnormal category prediction result and the abnormal category label represents the accuracy of the model to be trained, the text processing device calculates the loss function value based on the difference between the abnormal category prediction result and the abnormal category label.

[0123] In an embodiment of the present application, a text processing device determines a loss function value based on the difference between an abnormal category prediction result and an abnormal category label, including: the text processing device performs the following processing for the predicted probability of each abnormal category in the abnormal category prediction result: determining the abnormal label corresponding to the predicted probability from the abnormal category label, and calculating the category loss function value based on the difference between the predicted probability and the abnormal label; then, obtaining N category loss function values ​​corresponding to the N abnormal categories from the category loss function value of each abnormal category; finally, integrating the N category loss function values ​​into the loss function value.

[0124] It should be noted that the abnormal category prediction result includes N prediction probabilities, and the abnormal category label includes N abnormal labels. Here, the N prediction probabilities correspond to the N abnormal labels one by one. The text processing device calculates the sub-loss function value for each abnormal category, and obtains the category loss function value. For example, the text processing device can use at least one of the following loss functions to calculate the category loss function value: cross entropy loss function, mean square error loss function, logarithmic loss function, and binary classification loss function. In addition, the text processing device can integrate by weighted summing the N category loss function values.

[0125] Step 116: Train the model to be trained based on the loss function value to obtain a text detection model.

[0126] In an embodiment of the present application, the text processing device performs backpropagation in the model to be trained based on the calculated loss function value to adjust the model parameters in the model to be trained; in addition, the training of the model to be trained can be a single time or iterative; if iterative training is performed, then when the iterative training is completed, the model to be trained trained by the current iterative training is the text detection model.

[0127] It should be noted that when the text processing device determines that the iterative training meets the training end condition, the iterative training is determined to be ended; otherwise, the iterative training continues. The training end condition may be reaching the accuracy index threshold, reaching the iteration number threshold, reaching the iteration duration threshold, or a combination of the above, etc., which is not limited in the embodiments of the present application.

[0128] In the embodiments of the present application, when the text processing device is used to train a text detection model, it can be various servers; when the text processing device is used to respond to information generation requests, it can also be various servers or various terminals; and the text processing device can also be one electronic device or multiple electronic devices, etc., which is not limited in the embodiments of the present application.

[0129] The following will describe an exemplary application of the embodiment of the present application in a practical application scenario. The exemplary application describes a process of performing a priori quality detection on the prompt text in a text image scenario.

[0130] See also Figure 7 , Figure 7 is an exemplary Wensheng diagram flow chart provided in the embodiment of the present application; Figure 7 As shown, before using the Wensheng graph model 7-2 (AIGC model, such as Stable Diffusion, called text detection model) to process the prompt text 7-1 (called the text to be detected), the prompt text 7-1 is first input into the quality detection model 7-3 to perform quality detection on the prompt text 7-1 and determine possible problems in the prompt text 7-1, so that the prompt text 7-1 with problems can be corrected in time; or, when possible problems in the prompt text 7-1 are detected, the prompt text 7-1 is processed by using the Wensheng graph model 7-2 to obtain multiple images 7-4 (called third target information), and based on the detected problems, the corresponding modules to be detected are determined from multiple detection modules 7-5 (exemplarily showing the face quality detection module 7-51, the limb deformation detection module 7-52 and the clarity detection module 7-53, called modules for performing P types of abnormality detection), and then the multiple images 7-4 are input into the determined modules to be detected for image detection.

[0131] See also Figure 8 , Figure 8 is an exemplary prompt text quality detection model training diagram provided in an embodiment of the present application; Figure 8 As shown, the sample prompt text 8-1 (referred to as the sample text) is generated into multiple images 8-2 through the text-based graph model 7-2, and multiple detection modules 7-5 detect each of the multiple images 8-2 respectively to obtain image annotation data 8-3 (referred to as anomaly category labels); a quality detection model 7-3 is trained based on the image annotation data 8-3 and the sample prompt text 8-1.

[0132] It should be noted that in the text-image scenario in the advertising field, the text processing method provided in the embodiment of the present application includes label acquisition, model training and model application processes, and each process is described below.

[0133] When acquiring labels in the Vincent graph scenario in the advertising field, first, according to the needs of the image to be generated, the prompt text for generating the image is obtained; then, the prompt text is used to generate multiple images through the Vincent graph model, and the quality of the multiple images is inspected; finally, labels of multiple images are obtained based on the quality inspection results, such as qualified images, images with abnormal scene resolution, distorted images, images with abnormal morphology, etc.

[0134] See also Fig. 9 , Fig. 9 is an exemplary label schematic diagram provided in the embodiment of the present application; Fig. 9 As shown, image 9-1 is an image obtained by performing a text map based on prompt text 9-2. A quality inspection is performed on image 9-1, and a label of image 9-1 is determined to be label 9-3 based on the quality inspection result, that is, Figure 8 Image annotation data in 8-3.

[0135] When training the model, the quality detection of the prompt text is abstracted to train a quality detection model whose input is the prompt text of the text image and whose output is the abnormal category label.

[0136] based on Figure 8 , see Fig.10 , Fig.10 is an exemplary quality detection model provided in the embodiments of the present application; Fig.10 As shown, the quality inspection model 7-3 to be trained includes a preprocessing and encoding module 10-1, a natural language processing module 10-2 (for example, a large language model BERT) and a linear layer 10-3; the preprocessing and encoding (Tokenization) module 10-1 is used to preprocess the sample prompt text 8-1 to obtain a word segmentation sequence. Here, the sample prompt text is preprocessed accordingly according to different application scenarios (for example, unifying uppercase and lowercase letters, removing punctuation marks, word segmentation, etc.). Here, the sample prompt text 8-1 is taken as an example for explanation in Chinese; the word segmentation sequence is then encoded to obtain a coded sequence 10-4; then, the natural language processing module 10-2 is used to vectorize the coded sequence 10-4 to obtain a vector representation 10-5; finally, the linear layer 10-3 is used to fine-tune the vector representation 10-5 to obtain a multi-category classification result 10-6 (category 1, category 2, category 3, ..., category m), wherein the multi-category classification result 10-6 is used to obtain Figure 8 Image annotation data in 8-3.

[0137] It should be noted that the natural language processing module is composed of a stack of multi-layer bidirectional transformer models (Transformer models). It is a large language model (LLM) trained with multiple natural language processing tasks, which can convert the input encoding sequence into a vector. The linear layer is a fully connected neural network used to learn classification tasks from the features extracted from the natural language processing module; the fully connected network includes one or more hidden layers and an output layer. The output dimension is the number of abnormal categories, which can fit different classifications. In addition, the linear layer also includes a fully connected output layer, which outputs a probability distribution as a multi-category classification result.

[0138] For example, when the cross entropy loss function is used to calculate the loss value, it is as shown in formula (1).

[0139] L n =-[y n ·logσ(x n )+(1-y n )·log(1-σ(x n ))] (1);

[0140] Among them, L n is the loss value of the nth category (called the category loss function value), y n is the value corresponding to the true classification result of the nth category in the image annotation data (called the abnormal label), σ(x n ) is the probability corresponding to the nth category predicted by the quality detection model to be trained (called the predicted probability).

[0141] Here, the final loss function value can be obtained by accumulating the loss values ​​of each category, and the quality detection model can be trained based on the final loss function value.

[0142] In the process of model application, after the prompt text is input into the quality detection model, the potential problems of the image generated by the prompt text (called N abnormality probabilities corresponding to N abnormal categories) can be output, so that the decision of the prompt text can be determined for the potential problem; for example, as shown in Table 1.

[0143] Table 1

[0144]

[0145] It should be noted that when determining to correct the prompt text, after correcting the content of the prompt text, the corrected prompt text can be re-checked for quality, and when the confidence score of each abnormal category is lower than 0.5 (called the probability threshold), it is determined that the quality inspection has passed, and the subsequent Wensheng image processing can continue; or, Wensheng image processing is first performed based on the prompt text, and then the generated image is input into the quality detection model with the probability of the abnormal category greater than the probability threshold for detection.

[0146] It should be noted that for multiple prompt texts, the quality detection results of the prompt texts can be accurately determined based on the abnormal categories with the top K abnormal probabilities. Fig.11 , Fig.11 is a schematic diagram of an exemplary detection indicator trend provided in an embodiment of the present application; Fig.11As shown in the figure, as the number of K increases, the increasing trend of accuracy is shown in the broken line 11-1; it is easy to know that as the number of K increases, the accuracy is higher, and when K is 5, the accuracy reaches more than 0.90. Therefore, when it is determined that the probability of the abnormal category is greater than the probability threshold, the 5 detection modules corresponding to the abnormal categories with the top 5 abnormal probabilities are selected to detect the generated image, which reduces the number of detection categories, can improve detection efficiency, and reduce detection resource consumption.

[0147] It is understandable that when the text processing method provided in the embodiment of the present application is applied to the Wensheng graph scene, it is possible to make a priori judgment on the abnormal categories that may exist in the generated image corresponding to the current prompt text, provide effective information for downstream applications, and improve data processing efficiency. In addition, by making a priori judgment on the prompt text, the prompt text can be corrected in time to prompt the efficiency and quality of the generated image. And, by making a priori judgment on the prompt text, it is also possible to perform targeted detection on the generated image based on the detected abnormal category, which reduces the number of detection categories of the generated image, can reduce the detection consumption of the generated image, and improve the detection efficiency; and, by using the abnormal categories corresponding to the top K abnormal probabilities for evaluation, it is possible to reduce the detection consumption of the generated image and improve the detection efficiency. In addition, when the Wensheng graph is an advertising image generation scene, the quality and efficiency of the generation of the advertising image can be improved by the text processing method provided in the embodiment of the present application.

[0148] The following is a description of an exemplary structure of a text processing device 455 provided in an embodiment of the present application implemented as a software module. In some embodiments, Figure 2 As shown, the software modules stored in the text processing device 455 of the memory 450 may include:

[0149] The request response module 4551 is used to obtain the text to be detected in response to the information generation request, wherein the text to be detected is a prompt text, and the text to be detected is used to generate information of the target type;

[0150] A feature extraction module 4552 is used to encode the text to be detected into features to be detected;

[0151] Anomaly prediction module 4553, used to predict N anomaly probabilities corresponding to N anomaly categories based on the features to be detected, where N is a positive integer and the anomaly category refers to a problem category existing in the information of the target type;

[0152] The information generation module 4554 is used to generate the first target information based on the text to be detected when M of the abnormal probabilities among the N abnormal probabilities are less than their respective corresponding probability thresholds, wherein M≤N, and M is a positive integer, and the first target information is information of the target type.

[0153] In an embodiment of the present application, the text processing device 455 also includes an information correction module 4555, which is used to correct the text to be detected when L of the abnormal probabilities among the N abnormal probabilities are greater than or equal to the corresponding probability thresholds, so as to obtain a first prompt text, wherein L≤N, and L is a positive integer; and generate second target information based on the first prompt text, wherein the second target information is information of the target type.

[0154] In an embodiment of the present application, the information generation module 4554 is also used to determine P abnormal categories corresponding to the L abnormal probabilities when L of the abnormal probabilities among the N abnormal probabilities are greater than or equal to the corresponding probability thresholds, wherein P≤N and P is a positive integer; generate third target information based on the text to be detected, wherein the third target information is information of the target type; perform P types of abnormality detection corresponding to the P abnormal categories on the third target information; and when it is detected that the third target information does not contain information belonging to any of the P abnormal categories, determine the third target information as the final target information.

[0155] In an embodiment of the present application, the information generation module 4554 is also used to determine, when L is less than the minimum number of categories, L abnormal categories corresponding to the L abnormal probabilities, and determine the L abnormal categories as P abnormal categories; when L is greater than or equal to the minimum number of categories, select the abnormal probability of the minimum number of categories with the largest abnormal probability from the L abnormal probabilities, and determine the abnormal category of the minimum number of categories corresponding to the abnormal probability of the minimum number of categories as P abnormal categories.

[0156] In an embodiment of the present application, the information correction module 4555 is also used to correct the text to be detected to obtain a second prompt text when it is detected that the third target information contains information belonging to any of the P abnormal categories; and generate fourth target information based on the second prompt text, wherein the fourth target information is information of the target type.

[0157] In an embodiment of the present application, the feature extraction module 4552 is also used to preprocess the text to be detected to obtain the text to be encoded; segment the text to be encoded to obtain a sequence of segmented words to be encoded; encode each segmented word to be encoded in the sequence of segmented words to be encoded to obtain a segmented word encoding sequence; and perform vector conversion on the segmented word encoding sequence to obtain the feature to be detected.

[0158] In an embodiment of the present application, the N abnormal probabilities are obtained through a text detection model, and the text processing device 455 also includes a model training module 4556, which is used to obtain training data, wherein the training data includes sample text and abnormal category labels; based on the model to be trained, the abnormal category prediction result corresponding to the sample text is obtained, wherein the model to be trained is an artificial neural network model to be trained for detecting prompt text; based on the difference between the abnormal category prediction result and the abnormal category label, a loss function value is determined; based on the loss function value, the model to be trained is trained to obtain the text detection model.

[0159] In an embodiment of the present application, the model training module 4556 is also used to generate information for the sample text to obtain target sample information; perform anomaly detection on the target sample information to obtain the anomaly category label; and combine the sample text and the anomaly category label into the training data.

[0160] In an embodiment of the present application, the model training module 4556 is also used to perform the following processing for the predicted probability of each abnormal category in the abnormal category prediction result: determine the abnormal label corresponding to the predicted probability from the abnormal category label; calculate the category loss function value based on the difference between the predicted probability and the abnormal label; obtain multiple category loss function values ​​corresponding to N abnormal categories from the category loss function value of each abnormal category; and integrate multiple category loss function values ​​into the loss function value.

[0161] In an embodiment of the present application, the target type includes at least one of the following: image type, text type, voice type, and video type.

[0162] In an embodiment of the present application, when the target type is an image type, the N abnormality categories include at least one of the following: abnormal generation method, abnormal clarity, abnormal morphology, and abnormal scene type, wherein the abnormal generation method means that the generated image is non-native, the abnormal clarity means that the clarity of the generated image is lower than the clarity threshold, the abnormal morphology means that the distortion of the generated image is higher than the distortion threshold, and the abnormal scene type means that the boundary value between the foreground and background of the generated image is lower than the boundary threshold.

[0163] The embodiment of the present application provides a computer program product, which includes computer executable instructions or computer programs, and the computer executable instructions or computer programs are stored in a computer readable storage medium. The processor of the text processing device reads the computer executable instructions or computer programs from the computer readable storage medium, and the processor executes the computer executable instructions or computer programs, so that the text processing device performs the text processing method described above in the embodiment of the present application.

[0164] The present application embodiment provides a computer-readable storage medium in which computer executable instructions or computer programs are stored. When the computer executable instructions or computer programs are executed by a processor, the processor will be caused to execute the text processing method provided by the present application embodiment, for example, Figure 3 The text processing method shown.

[0165] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0166] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0167] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0168] As an example, computer-executable instructions may be deployed to be executed on one electronic device (in which case, the one electronic device is a text processing device), or on multiple electronic devices located at one location (in which case, the multiple electronic devices located at one location are text processing devices), or on multiple electronic devices distributed at multiple locations and interconnected by a communication network (in which case, the multiple electronic devices distributed at multiple locations and interconnected by a communication network are text processing devices).

[0169] It is understandable that in the embodiments of the present application, when the embodiments of the present application are applied to specific products or technologies, it is necessary to obtain user permission or consent, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions. In addition, in the present application, the acquisition of information such as training data involves the implementation of data capture technology solutions. When the above embodiments of the present application are applied to specific products or technologies, the collection, use and processing of relevant data should comply with the requirements of national laws and regulations, comply with the principles of legality, legitimacy and necessity, do not involve the acquisition of data types prohibited or restricted by laws and regulations, and will not hinder the normal operation of the target website.

[0170] In summary, before generating information of the target type based on the text to be detected, the embodiment of the present application first detects the text to be detected to obtain N abnormal probabilities corresponding to N abnormal categories; since the target type of information is generated only when the M abnormal probabilities are less than the probability thresholds corresponding to each of them, it is ensured that the information generation is performed after the text to be detected passes the detection, and the information generation can be performed accurately to improve the quality of information generation, thereby improving the efficiency of information generation. In addition, when it is determined that the quality test of the text to be detected has not passed, the text to be detected can be corrected in time to improve the quality of information generation; or the generated information can be targetedly detected to reduce the text detection consumption and improve the text detection efficiency; and, the minimum number of categories with the highest abnormal probability is used to reduce the abnormal category detection amount of the generated information, which can reduce the detection consumption of the generated information and improve the detection efficiency of the generated information.

[0171] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A text processing method, characterized in that: The method comprises: In response to the information generation request, obtaining a text to be detected, wherein the text to be detected is a prompt text, and the text to be detected is used to generate information of a target type; Encoding the text to be detected as features to be detected; Based on the features to be detected, predict N abnormal probabilities corresponding to N abnormal categories, where N is a positive integer, and the abnormal category refers to the problem category existing in the information of the target type; When M of the abnormal probabilities among the N abnormal probabilities are less than their respective corresponding probability thresholds, first target information is generated based on the text to be detected, where M≤N and M is a positive integer, and the first target information is information of the target type.

2. The method according to claim 1, characterized in that After predicting N abnormality probabilities corresponding to N abnormality categories based on the features to be detected, the method further includes: When L of the abnormal probabilities among the N abnormal probabilities are greater than or equal to the corresponding probability thresholds, the text to be detected is corrected to obtain a first prompt text, wherein L≤N, and L is a positive integer; Second target information is generated based on the first prompt text, wherein the second target information is information of the target type.

3. The method according to claim 1, characterized in that After predicting N abnormality probabilities corresponding to N abnormality categories based on the features to be detected, the method further includes: When L of the abnormal probabilities among the N abnormal probabilities are greater than or equal to the probability thresholds corresponding to each other, determining P abnormal categories corresponding to the L abnormal probabilities, where P≤N, and P is a positive integer; Generating third target information based on the text to be detected, wherein the third target information is information of the target type; Performing P types of anomaly detection corresponding to the P anomaly categories on the third target information; When it is detected that the third target information does not contain information belonging to any of the P abnormal categories, the third target information is determined as the final target information.

4. The method according to claim 3, characterized in that The determining of the P abnormality categories corresponding to the L abnormality probabilities includes: When L is less than the minimum number of categories, determining L abnormal categories corresponding to the L abnormal probabilities, and determining the L abnormal categories as P abnormal categories; When L is greater than or equal to the minimum number of categories, the abnormal probability of the minimum number of categories with the largest abnormal probability is selected from the L abnormal probabilities, and the abnormal categories of the minimum number of categories corresponding to the abnormal probability of the minimum number of categories are determined as P abnormal categories.

5. The method according to claim 3, characterized in that: After performing P types of anomaly detection corresponding to the P anomaly categories on the third target information, the method further includes: When it is detected that the third target information contains information belonging to any of the P abnormal categories, the text to be detected is corrected to obtain a second prompt text; Fourth target information is generated based on the second prompt text, wherein the fourth target information is information of the target type.

6. The method according to any one of claims 1 to 5, characterized in that: The step of encoding the text to be detected into features to be detected includes: Preprocessing the text to be detected to obtain a text to be encoded; Segmenting the text to be encoded to obtain a sequence of segmented words to be encoded; Encode each to-be-encoded word in the to-be-encoded word sequence to obtain a word encoding sequence; The word segmentation coding sequence is vectorized to obtain the feature to be detected.

7. The method according to any one of claims 1 to 5, characterized in that: The process of obtaining the N abnormal probabilities based on the text to be detected is implemented by a text detection model, wherein the text detection model is trained by the following steps: Acquire training data, wherein the training data includes sample text and abnormal category labels; Obtaining an abnormal category prediction result corresponding to the sample text based on the model to be trained, wherein the model to be trained is an artificial neural network model to be trained for detecting the prompt text; Determining a loss function value based on a difference between the abnormal category prediction result and the abnormal category label; The model to be trained is trained based on the loss function value to obtain the text detection model.

8. The method according to claim 7, characterized in that The obtaining of training data comprises: Generate information on the sample text to obtain target sample information; Performing anomaly detection on the target sample information to obtain the anomaly category label; The sample text and the abnormal category label are combined into the training data.

9. The method according to claim 7, characterized in that: The determining of the loss function value based on the difference between the abnormal category prediction result and the abnormal category label includes: For the predicted probability of each abnormal category in the abnormal category prediction result, the following processing is performed: Determining, from the abnormal category labels, an abnormal label corresponding to the predicted probability; Calculating a category loss function value based on a difference between the predicted probability and the abnormal label; Obtaining a plurality of the category loss function values ​​corresponding to the N category abnormal categories from the category loss function value of each of the abnormal categories; A plurality of the category loss function values ​​are integrated into the loss function value.

10. The method according to any one of claims 1 to 5, characterized in that: The target type includes at least one of the following: image type, text type, voice type, and video type.

11. The method according to any one of claims 1 to 5, characterized in that: When the target type is an image type, the N abnormal categories include at least one of the following: abnormal generation method, abnormal clarity, abnormal morphology, and abnormal scene type, wherein the abnormal generation method means that the generated image is non-native, the abnormal clarity means that the clarity of the generated image is lower than the clarity threshold, the abnormal morphology means that the distortion of the generated image is higher than the distortion threshold, and the abnormal scene type means that the boundary value between the foreground and background of the generated image is lower than the boundary threshold.

12. A text processing device, characterized in that: The text processing device comprises: A request response module, used for obtaining a text to be detected in response to an information generation request, wherein the text to be detected is a prompt text, and the text to be detected is used to generate information of a target type; A feature extraction module, used for encoding the text to be detected into features to be detected; An anomaly prediction module, used to predict N anomaly probabilities corresponding to N anomaly categories based on the features to be detected, where N is a positive integer and the anomaly category refers to a problem category existing in the information of the target type; An information generation module is used to generate first target information based on the text to be detected when M of the N abnormal probabilities are less than their respective corresponding probability thresholds, wherein M≤N, and M is a positive integer, and the first target information is information of the target type.

13. An electronic device for text processing, characterized in that: The electronic device comprises: A memory for storing computer executable instructions or computer programs; A processor, configured to implement the text processing method according to any one of claims 1 to 11 when executing the computer executable instructions or computer programs stored in the memory.

14. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer programs are executed by a processor, the text processing method according to any one of claims 1 to 11 is implemented.

15. A computer program product comprising computer executable instructions or a computer program, characterized in that: When the computer executable instructions or computer programs are executed by a processor, the text processing method according to any one of claims 1 to 11 is implemented.

Images