Text processing method and device, equipment, storage medium and program product

Abstract

The invention relates to a text processing method and device, equipment, a storage medium and a program product, and relates to the technical field of artificial intelligence. The method comprises the following steps: acquiring a first input text, wherein the first input text is an input text processed by a natural language; obtaining compliance identification information of the first input text, wherein the compliance identification information is used for indicating whether the first input text is compliant or not; under the condition that the compliance identification information indicates that the first input text is not compliant, obtaining a first output result corresponding to the first input text based on a compliance database; and compliance candidate output results are stored in the compliance database. According to the method and the device, the compliant output result can be output in time under the condition that the input text is not compliant, and the interaction efficiency when a user uses a natural language processing system is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a text processing method, apparatus, device, storage medium, and program product. Background Technology

[0002] Natural language processing systems can perform natural language processing on input text and generate corresponding output content (such as text, images, etc.).

[0003] If a natural language processing system processes non-compliant input text, it may output non-compliant results. In related technologies, in order to ensure the ecological security of generative language models, natural language processing systems usually refuse to respond to non-compliant input text.

[0004] However, in the solutions of related technologies, users often need to repeatedly modify the input text that is identified as non-compliant before it can be processed normally, which affects the interaction efficiency of users when using natural language processing systems. Summary of the Invention

[0005] This application provides a text processing method, apparatus, device, storage medium, and program product that can promptly output compliant results when the input text is non-compliant, thereby improving the interactive efficiency of users when using a natural language processing system. The technical solution is as follows:

[0006] According to one aspect of this application, a text processing method is provided, the method comprising:

[0007] Obtain the first input text, which is the input text for natural language processing;

[0008] Obtain compliance identification information of the first input text, wherein the compliance identification information is used to indicate whether the first input text is compliant;

[0009] If the compliance identification information indicates that the first input text is non-compliant, a first output result corresponding to the first input text is obtained based on the compliance database; the compliance database stores compliant candidate output results.

[0010] According to one aspect of this application, a text processing apparatus is provided, the apparatus comprising:

[0011] The input acquisition module is used to acquire the first input text, which is the input text for natural language processing.

[0012] The information acquisition module is used to acquire compliance identification information of the first input text, wherein the compliance identification information is used to indicate whether the first input text is compliant;

[0013] The output acquisition module is used to acquire a first output result corresponding to the first input text based on a compliance database when the compliance identification information indicates that the first input text is non-compliant; the compliance database stores compliant candidate output results.

[0014] In some embodiments, the candidate output results in the compliance database correspond to reference input text; the output acquisition module is configured to:

[0015] When the compliance identification information indicates that the first input text is non-compliant, relevant information is obtained, including a first correlation value and a second correlation value; the first correlation value is used to indicate the correlation between the first input text and the first reference input text, and the second correlation value is used to indicate the correlation between the first input text and the first candidate output result; the first candidate output result is a candidate output result in the database, and the first candidate output result corresponds to the first reference input text;

[0016] Based on the aforementioned relevant information, the matching degree between the first input text and the first candidate output result is obtained;

[0017] If the matching degree between the first input text and the first candidate output result meets the matching condition, the first output result is obtained based on the first candidate output result.

[0018] In some embodiments, the output acquisition module is configured to:

[0019] Based on the first candidate output result, modify the first input text to obtain the second input text;

[0020] The second input text is passed to the generative language model that performs the natural language processing, and the first output result returned by the generative language model is obtained.

[0021] In some embodiments, the output acquisition module is configured to:

[0022] Obtain a first description text, which is used to describe the first candidate output result;

[0023] Add the first description text to the first input text to obtain the second input text.

[0024] In some embodiments, the relevant information further includes a third correlation value, which is used to indicate the correlation between the first candidate output result and the first reference input text;

[0025] The output acquisition module is used to obtain the matching degree between the first input text and the first candidate output result based on the first correlation value, the second correlation value and the third correlation value.

[0026] In some embodiments, the output acquisition module is configured to:

[0027] The first correlation value, the second correlation value, and the third correlation value are weighted to obtain a weighted processing result.

[0028] Based on the weighted processing result, the matching degree between the first input text and the first candidate output result is obtained.

[0029] In some embodiments, the apparatus further includes: a text generation module, configured to:

[0030] If no matching output result exists in the compliance database, a second candidate output result is obtained. The second candidate output result is the candidate output result with the highest matching degree with the first input text in the compliance database. The matching output result is the candidate output result whose matching degree with the first input text satisfies the matching condition.

[0031] Obtain a second description text, which is used to describe the second candidate output result;

[0032] Based on the second description text, generate the third input text;

[0033] The third input text is passed to the generative language model that performs the natural language processing to obtain the second output result returned by the generative language model;

[0034] If the matching degree between the first input text and the second output result is greater than the matching degree threshold, the second output result is determined as the first output result.

[0035] In some embodiments, the apparatus further includes: a text editing module, configured to:

[0036] If no matching output result exists in the compliance database, the first input text is modified to obtain a compliant fourth input text; the matching output result is a candidate output result whose matching degree with the first input text satisfies the matching condition.

[0037] The fourth input text is passed to the generative language model that performs the natural language processing to obtain the third output result returned by the generative language model.

[0038] In some embodiments, if the compliance identification information indicates that the first input text is non-compliant, the compliance identification information is also used to indicate the risk level of the first input text;

[0039] The output acquisition module is used to acquire a first output result corresponding to the first input text based on the compliance database when the risk level of the first input text is a specified risk level.

[0040] In some embodiments, the information acquisition module is configured to:

[0041] Extract keywords from the first input text;

[0042] Input the first input text and the keywords into the compliance identification model to obtain the compliance identification information output by the compliance identification model;

[0043] The compliance identification model is a machine learning model trained based on input text samples and annotation information; the annotation information is used to indicate whether the input text sample is compliant.

[0044] In some embodiments, the information acquisition module is configured to:

[0045] Perform word segmentation on the first input text to obtain each candidate word in the first input text;

[0046] The candidate words that exist in the compliance check terminology library are extracted as the keywords; the compliance check terminology library contains words that have the risk of non-compliance.

[0047] According to another aspect of this application, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one computer instruction, the at least one computer instruction being loaded and executed by the processor to implement the text processing method as described above.

[0048] According to another aspect of this application, a computer-readable storage medium is provided, wherein at least one computer instruction is stored therein, the at least one computer instruction being loaded and executed by a processor to implement the text processing method as described above.

[0049] According to another aspect of this application, a computer program product is provided, the computer program product including computer instructions stored in a computer-readable storage medium, wherein a processor reads from the computer-readable storage medium and executes the computer instructions to implement the text processing method described above.

[0050] The technical solutions provided in this application embodiment may have the following beneficial effects:

[0051] The solution shown in this application embodiment, for a first input text that needs to be processed by natural language, when the first input text is non-compliant, the computer device can obtain a compliant first output result corresponding to the first input text from the compliance database; the above solution can automatically obtain a compliant first output result from the compliant candidate output results and feed it back to the user when the first input text is non-compliant, without requiring the user to repeatedly modify the input text, thereby improving the interaction efficiency of the user when using the natural language processing system. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is an architectural diagram of a computer system provided in an exemplary embodiment of this application;

[0054] Figure 2 This is a flowchart of a text processing method provided in an exemplary embodiment of this application;

[0055] Figure 3 This is a flowchart of a text processing method provided in an exemplary embodiment of this application;

[0056] Figure 4 This is a schematic diagram illustrating the training and inference of a multi-model dual-tower structure matching model provided in an exemplary embodiment of this application;

[0057] Figure 5 This is a flowchart illustrating the process of obtaining descriptive text provided in an exemplary embodiment of this application;

[0058] Figure 6 This is a flowchart of a text processing method provided in an exemplary embodiment of this application;

[0059] Figure 7 This is a schematic diagram illustrating the training and inference of a compliance identification model provided in an exemplary embodiment of this application;

[0060] Figure 8 This is a flowchart of a text processing method provided in an exemplary embodiment of this application;

[0061] Figure 9 This is a flowchart of a prompt word processing method provided in an exemplary embodiment of this application;

[0062] Figure 10 This is a flowchart of a method for optimizing the content ecosystem security of a large language model provided in an exemplary embodiment of this application;

[0063] Figure 11 This is a schematic diagram of an iterative learning method provided in an exemplary embodiment of this application;

[0064] Figure 12 This is a schematic diagram of a decision boundary sampling method provided in an exemplary embodiment of this application;

[0065] Figure 13 This is a flowchart of a query text processing method provided in an exemplary embodiment of this application;

[0066] Figure 14 This is a block diagram illustrating a text processing apparatus according to an exemplary embodiment of this application;

[0067] Figure 15 This is a structural block diagram of a computer device provided in an exemplary embodiment of this application.

[0068] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application. Detailed Implementation

[0069] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0070] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0071] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0072] In this application embodiment, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions. For example, the attack operations and other target behaviors involved in this application are all obtained under full authorization.

[0073] It should be understood that although the terms first, second, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, a first parameter may also be referred to as a second parameter without departing from the scope of this disclosure, and similarly, a second parameter may also be referred to as a first parameter. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0074] The following is a definition of some terms used in this application:

[0075] 1) Artificial Intelligence (AI): This refers to the theories, methods, technologies, and application systems that utilize digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce new intelligent machines that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities.

[0076] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, pre-trained model technology, operating / interactive systems, and mechatronics. Among these, pre-trained models, also known as large-scale models or foundational models, can be widely applied to downstream tasks across various AI fields after fine-tuning. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0077] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, digital twins, virtual humans, robots, AI-generated content (AIGC), conversational interaction, smart healthcare, smart customer service, and game AI. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0078] 2) Natural Language Processing (NLP): This is an important area within computer science and artificial intelligence. It studies the theories and methods for enabling effective communication between humans and computers using natural language. NLP involves natural language, the language people use in daily life, and is closely related to linguistics research; it also involves computer science and mathematics, and is a crucial technique for model training in artificial intelligence. Pre-trained models evolved from Large Language Models (LLMs) in NLP. After fine-tuning, large language models can be widely applied to downstream tasks. NLP techniques typically include text processing, semantic understanding, machine translation, question answering, and knowledge graphs.

[0079] 3) Machine Learning (ML): This is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instruction-based learning. Pre-trained models are the latest development in deep learning, integrating the above techniques.

[0080] 4) Artificial Intelligence Cloud Services: Also known as AI as a Service (AIaaS). This is currently a mainstream service model for artificial intelligence platforms. Specifically, AIaaS platforms break down several common AI services and provide them as independent or packaged services in the cloud. This service model is similar to opening an AI-themed marketplace: all developers can access and use one or more AI services provided by the platform through API interfaces. Some experienced developers can also use the AI ​​framework and AI infrastructure provided by the platform to deploy and maintain their own dedicated cloud AI services.

[0081] 5) AI Generated Content (AIGC): Advanced AI models with broad understanding and learning capabilities perform complex tasks and generate creative content in multiple fields; such as text-to-text, text-to-image, text-to-video, text-to-3D, and other AIGC application scenarios.

[0082] 6) Perceiver Model: This is a general-purpose perceiver architecture capable of handling diverse input data. Based on self-attention and cross-attention mechanisms, the perceiver model effectively captures complex patterns in the input data. It consists of multiple layers, each containing a self-attention sublayer and a cross-attention sublayer. The self-attention sublayer allows the model to focus on different parts of the input data, while the cross-attention sublayer allows the model to pass information between different layers. The perceiver model can be applied to natural language processing tasks such as text classification and sentiment analysis. By segmenting text into a series of words or symbols and processing these words or symbols using the perceiver model, the topic or sentiment of the text can be effectively identified. In natural language processing applications, a pre-trained perceiver model can be used as a feature extractor, followed by a classifier for natural language classification.

[0083] 7) Large Language Model (LLM): Also known as a large language model, it is a type of AI model. An LLM can automatically generate corresponding output results based on the natural language text input to the LLM, and the output results can be in various forms such as text and images.

[0084] Prompts, in the field of pre-trained large-scale language models, are a form of input used to instruct AI models on what actions to take or what outputs to generate when performing a specific task. A prompt is a natural language input, similar to a command or instruction, letting the AI ​​model know what it needs to do. In other words, in an interactive system based on generative language models, the user inputs a prompt, and the system responds accordingly.

[0085] With the widespread application of large-scale language models (LLMs) across various industries, the ecological security issues of LLMs are becoming increasingly prominent. When answering questions, large-scale language models may involve multiple aspects such as ethics and laws and regulations. If not properly handled, these issues could have adverse effects on society and individuals. Therefore, to address the ecological security issues of large-scale language models, the solutions illustrated in the following embodiments of this application can be adopted to ensure that the content generated by LLMs complies with ethical, value, and legal norms.

[0086] The solutions provided in this application involve artificial intelligence technologies such as natural language processing and machine learning, which are specifically illustrated through the following embodiments.

[0087] Please refer to Figure 1 This illustration shows an architectural diagram of a computer system provided in an exemplary embodiment of this application. This computer system can be implemented as an architecture for a text processing configuration system. Figure 1 As shown, the computer system may include a terminal device 100 and a server 200. The terminal device 100 and the server 200 may be directly or indirectly connected via wired or wireless communication, and this application does not impose any restrictions on this connection.

[0088] Optionally, the terminal device 100 may be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. A client application for the target application may be installed and run on the terminal device 100. This target application may be an application with natural language processing capabilities, such as a generative language model (e.g., a large language model). This application does not limit the form of the target application, including but not limited to applications (Apps), mini-programs, etc., installed on the terminal device 100, and may also be in the form of a webpage.

[0089] Optionally, server 200 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services such as cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and basic cloud computing services such as big data and artificial intelligence platforms. Among these, the cloud server for big data and artificial intelligence platforms can provide artificial intelligence cloud services. Server 200 can be the backend server for the aforementioned target application, used to provide backend services to the client of the target application. For example, consider the application of text processing methods in an AI dialogue scenario:

[0090] Terminal device 100 can send a first input text 101 requiring text processing to server 200 through an interface provided by server 200; server 200 performs text processing on the first input text 101. Optionally, server 200 obtains compliance identification information 102 of the first input text 101. When compliance identification information 102 indicates that the first input text 101 is compliant, server 200 generates a first output text 103 corresponding to the first input text 101 through natural language processing; then, server 200 sends the first output text 103 to terminal device 100. Specifically, for example, if the first input text 101 is "What are the representative works of the poet Li Bai?", the first output text 103 could be "The representative works of the poet Li Bai include 'Looking at the Waterfall at Mount Lu,' 'The Road to Shu is Hard,' 'Drinking Alone Under the Moon,' and 'Departing from Baidi City at Dawn,' etc." When compliance identification information 102 indicates that the first input text 101 is non-compliant, server 200 obtains a second output text 105 corresponding to the first input text 101 based on compliance database 104. Then, server 200 sends the aforementioned second output text 105 to terminal device 100.

[0091] Please refer to Figure 2 The diagram illustrates a flowchart of a text processing method provided in an exemplary embodiment of this application. The method is executed by a computer device, which may optionally be a computer device... Figure 1 The system shown can be a terminal device 100; or it can be a server 200, or it can be both a terminal device 100 and a server 200. Figure 2 As shown, the method may include steps 210, 220 and 230.

[0092] Step 210: Obtain the first input text, which is the input text for natural language processing.

[0093] The computer device performs natural language processing on the first input text and then obtains the output result of the first input text. The output result is a compliant result that conforms to the first input text, and the output result may include, but is not limited to, one or more of the following forms: text, image, video, and animation.

[0094] For example, the first input text mentioned above could be a prompt word entered by the user and processed by a generative language model.

[0095] For example, the first input text may be text entered by a user of the terminal device 100; correspondingly, the output result may be a compliance result corresponding to the first input text, output by the server 200.

[0096] Step 220: Obtain compliance identification information of the first input text. The compliance identification information is used to indicate whether the first input text is compliant.

[0097] The aforementioned compliance identification information may be information related to the compliance of the first input text. Whether the first input text is compliant may refer to whether the first input text meets requirements regarding language norms, content review, data privacy, laws and regulations, and ethical standards.

[0098] In this embodiment, the computer device can perform compliance recognition processing on the first input text to obtain compliance recognition information of the first input text; that is, the computer device can determine whether the first input text is compliant or non-compliant through compliance recognition processing. For example, the computer device can use natural language processing (NLP) technology to perform word segmentation, part-of-speech tagging, entity recognition, sentiment analysis, and other processing on the first input text to extract features related to compliance / non-compliance; then, according to the compliance recognition rules set by the developer user, the compliance recognition information of the first input text can be obtained.

[0099] For example, a computer device can input the aforementioned first input text into a compliance recognition model. The compliance recognition model performs compliance recognition processing on the first input text and outputs compliance recognition information. For instance, the compliance recognition information output by the compliance recognition model can be result 0 or result 1, where result 0 represents that the first input text is compliant, and result 1 represents that the first input text is non-compliant. Optionally, result 1 can also include non-compliance types such as result 10 and result 11. For example, result 10 represents that the first input text does not comply with data privacy requirements, and result 11 represents that the first input text does not comply with legal and regulatory requirements. The compliance recognition model is a machine learning model trained based on input text samples and annotation information, where the annotation information is used to indicate whether the input text sample is compliant.

[0100] Step 230: If the compliance identification information indicates that the first input text is non-compliant, obtain the first output result corresponding to the first input text based on the compliance database; the compliance database stores compliant candidate output results.

[0101] In this embodiment of the application, if the conclusion that the first input text is non-compliant is reached after the above compliance identification process, the computer device can query the candidate output results corresponding to the first input text in the compliance database, and determine the first output result of the first input text based on the candidate output results corresponding to the first input text.

[0102] The aforementioned compliance database can be a database consisting of multiple non-compliant reference input texts and compliant candidate output results corresponding to each reference input text.

[0103] For example, when compliance identification information indicates that the first input text is non-compliant, the above-mentioned acquisition of the first output result corresponding to the first input text based on the compliance database can be implemented in the following steps: the computer device can traverse each reference input text in the above-mentioned compliance database and obtain the semantic similarity between the first input text and each reference input text; the computer device obtains a reference input text corresponding to the highest semantic similarity and the candidate output result corresponding to the reference input text; the computer device uses the candidate output result as the first output result of the first input text.

[0104] For example, developers can pre-configure the aforementioned compliance database on their computer devices. For instance, they can collect and organize several non-compliant reference input texts to form a reference input text set. This set can cover various potentially non-compliant input texts, as well as various possible language variations and veiled expressions. For each non-compliant reference input text, one or more corresponding compliant candidate output results are provided, forming a candidate input result set. This candidate input result set meets requirements related to language specifications, content moderation, data privacy, laws and regulations, and ethical standards. Optionally, the candidate output results can be text, images, etc., or response strategies (such as triggering manual intervention, terminating the dialogue, etc.).

[0105] In some embodiments, when compliance identification information indicates that the first input text is compliant, the computer device can pass the first input text to a large language model and obtain the output of the large language model.

[0106] For example, the first input text can be the input text of a large language model. Accordingly, if the compliance recognition information indicates that the first input text is compliant, the output result can be automatically generated by the large language model based on the input text and the task instructions for the input text. The task instructions include, but are not limited to, one or more of the following: translating text, extracting summaries, analyzing text, answering questions, creating text, and drawing images.

[0107] For example, the first input text can also be the query text of the search system. Accordingly, if the compliance identification information indicates that the first input text is compliant, the output results can be collected by the search system from the Internet or a database.

[0108] In summary, the solution shown in this application embodiment allows the computer device to obtain a compliant first output result corresponding to the first input text from a compliant database when the first input text is non-compliant. This solution can automatically obtain a compliant first output result from compliant candidate output results and provide it to the user when the first input text is non-compliant, eliminating the need for the user to repeatedly modify the input text and thus improving the user's interaction efficiency when using the natural language processing system.

[0109] Based on the above Figure 2 In one possible implementation of the scheme in the illustrated embodiment, the candidate output results in the compliance database correspond to reference input text.

[0110] In other words, the compliance database stores multiple candidate outputs for compliance, each corresponding to one or more reference input texts. For example, the compliance database could be a question-and-answer database consisting of reference input texts and corresponding candidate outputs for compliance.

[0111] Please refer to Figure 3 This illustrates a flowchart of a text processing method provided in an exemplary embodiment of this application. Figure 3 As shown above, Figure 2 Step 230 in the illustrated embodiment can be implemented as steps 230a, 230b, and 230c.

[0112] Step 230a: If the compliance identification information indicates that the first input text is non-compliant, obtain relevant information, including a first correlation value and a second correlation value; the first correlation value is used to indicate the correlation between the first input text and the first reference input text, and the second correlation value is used to indicate the correlation between the first input text and the first candidate output result; the first candidate output result is a candidate output result in the compliance database, and the first candidate output result corresponds to the first reference input text.

[0113] For example, when the first input text is non-compliant, the computer device can perform a relevance match between the first input text and each reference input text in the compliance database to obtain various first relevance values; the higher the first relevance value, the higher the relevance between the first input text and the reference input text. Simultaneously, the computer device can perform a relevance match between the first input text and each candidate output result in the compliance database to obtain various second relevance values; the higher the second relevance value, the higher the relevance between the first input text and the candidate output result.

[0114] For example, taking text as the candidate output, the first relevance value mentioned above can be the semantic similarity between the first input text and the reference input text. The higher the semantic similarity, the more closely related the first input text and the reference input text are. The second relevance value mentioned above can be the semantic similarity between the first input text and the candidate output. The higher the semantic similarity, the more logically consistent the first input text and the candidate output are.

[0115] The computer device can use a pre-trained word vector model to convert the combination of the first input text and each reference input text into vector pairs. Then, it calculates the cosine similarity, Euclidean distance, etc., between the vectors in each vector pair to obtain the semantic similarity between the first input text and each reference input text. Similarly, the computer device converts the combination of the first input text and each candidate output result into vector pairs. Then, it calculates the cosine similarity, Euclidean distance, etc., between the vectors in each vector pair to obtain the semantic similarity between the first input text and each candidate output result.

[0116] Step 230b: Based on relevant information, obtain the matching degree between the first input text and the first candidate output result.

[0117] The higher the matching degree between the first input text and the first candidate output result, the closer the first candidate output result is to the compliant output result corresponding to the first input text.

[0118] For example, the matching degree between the first input text and the first candidate output result can be the sum of the first relevance value and the second relevance value, or it can be the weighted average of the first relevance value and the second relevance value; wherein, the weights of the first relevance value and the second relevance value can be preset by the developer.

[0119] Step 230c: If the matching degree between the first input text and the first candidate output result meets the matching condition, obtain the first output result based on the first candidate output result.

[0120] Developers can pre-set the matching conditions in their computer devices. For example, the matching conditions can be a matching range. When the matching degree between the first input text and the first candidate output result falls within the matching range, the computer device can obtain the first output result based on the first candidate output result.

[0121] For example, the process of obtaining the first output result based on the first candidate output result can be implemented as follows: the computer device directly determines the first candidate output result as the first output result corresponding to the first input text; or, the process of obtaining the first output result based on the first candidate output result can also be implemented as follows: the computer device modifies the first input text according to the first candidate output result, transmits the modified first input text to a large language model, and obtains the output result corresponding to the modified first input text.

[0122] Based on the above embodiments, this embodiment illustrates how to obtain a first output result corresponding to a first input text based on a compliance database. Specifically, it may include: when compliance identification information indicates that the first input text is non-compliant, the computer device can obtain a first correlation value between the first input text and a first reference input text, and a second correlation value between the first input text and a first candidate output result; when the first correlation value and the second correlation value meet preset conditions, the computer device can use the first candidate output result as the first output result. The above scheme considers both the correlation between the first input text and the first reference input text, as well as the correlation between the first input text and the first candidate output result, which can improve the accuracy of matching between the first candidate output result and the first input text, thereby ensuring the accuracy of the first output result.

[0123] Based on the solutions shown in the above embodiments of this application, in one possible implementation, step 230c can be implemented as step 230c1 and step 230c2.

[0124] Step 230c1: Based on the first candidate output result, modify the first input text to obtain the second input text.

[0125] The modifications to the first input text mentioned above may include, but are not limited to, text correction and information supplementation.

[0126] For example, the aforementioned text correction can be used to correct erroneous words, inaccurate words, or non-compliant statements in the first input text. For instance, after performing grammatical analysis and non-compliant word analysis on the first input text and concluding that the first input text contains factual errors, spelling errors, grammatical errors, vocabulary misuse, or non-compliant statements, the computer device can directly correct the errors or non-compliant statements in the first input text to ensure its accuracy.

[0127] For example, the above-mentioned information supplementation can be achieved by adding missing information to the first input text. For instance, by comparing and analyzing the first candidate output result with the first input text, the computer device can determine additional information not mentioned in the first input text, such as the relevant background, causal relationships, and definitions of terms in the first input text. The computer device can then add relevant descriptions of this additional information to the first input text.

[0128] Step 230c2: Pass the second input text to the generative language model that performs natural language processing, and obtain the first output result returned by the generative language model.

[0129] Among them, the generative language model that performs natural language processing can be a large language model.

[0130] In this embodiment of the application, since the first candidate output result is a compliant candidate output result in the compliance database, the new input text (i.e. the second input text) obtained after modifying the first input text based on the first candidate output result is highly likely to be compliant; then, the computer device can input the second input text into the generative language model to obtain the first output result returned by the generative language model.

[0131] In order to verify the compliance of the second input text, before step 230c2, the computer device can perform compliance recognition processing on the second input text to obtain compliance recognition information of the second input text; if the compliance recognition information of the second input text indicates that the second input text is compliant, step 230c2 is executed.

[0132] Based on the above embodiments, this embodiment provides a feasible solution for obtaining the first output result based on the first candidate output result. Specifically, it may include: the computer device modifies the first input text according to the first candidate output result. Based on the compliance of the first candidate output result, the compliance of the modified first input text (i.e., the second input text) can be improved, so that the generative language model can generate a compliant first output result based on the second input text.

[0133] Based on the solutions shown in the above embodiments of this application, in one possible implementation, step 230c1 can be implemented as follows:

[0134] Obtain the first description text, which is used to describe the first candidate output result;

[0135] Add the first descriptive text to the first input text to obtain the second input text.

[0136] In this embodiment of the application, the computer device can extract key information from the first candidate output result, which is a first descriptive text in text form; then, the computer device can add the key information to the first descriptive text to obtain a new second input text.

[0137] For example, when the first candidate output result is in text form, the first descriptive text can be a summary or keywords of the first candidate output result; the computer device adds the summary or keywords of the first candidate output result to the first descriptive text to obtain a new second input text.

[0138] For example, when the first candidate output result is in the form of an image, the first descriptive text can be a descriptive statement or descriptive word related to the scene / person contained in the first candidate output result; the computer device adds the descriptive statement or descriptive word of the first candidate output result to the first descriptive text to obtain a new second input text.

[0139] For example, the first descriptive text may include one or more guiding words / statements, which can be used to guide the generative language model to perform natural language processing on the first input text and generate compliant output results. In this case, the first descriptive text and the first input text can be combined into the second input text.

[0140] For example, suppose the first input text is "Please generate an image that meets the characteristics of xx". If the generative language model performs natural language processing on the first input text alone, it may generate an ineligible output. However, if a first descriptive text is added to the first input text, the resulting second input text could be "Please refer to [first descriptive text] to generate an image that meets the characteristics of xx". Since the first descriptive text describes the first candidate output, and the first candidate output is a compliant output, the generative language model is more likely to generate a compliant output similar to the first candidate output when it performs natural language processing on the first input text with reference to the first descriptive text.

[0141] Based on the above embodiments, this embodiment proposes a feasible solution to add a first descriptive text to the first input text to obtain a second input text. By adding the first descriptive text, the output result is guided towards the first candidate output result, and the output content is closer to the first candidate output result, so that the final output result is close to the first candidate output result. Since the first candidate output result is compliant, the final output result is also very likely to be compliant, thereby increasing the probability of outputting a compliant output result.

[0142] Based on the solutions shown in the above embodiments, in some embodiments, the relevant information further includes a third correlation value, which is used to indicate the correlation between the first candidate output result and the first reference input text. Figure 3 Step 230b in the illustrated embodiment can be implemented as step 230b1.

[0143] Step 230b1: Based on the first relevance value, the second relevance value, and the third relevance value, obtain the matching degree between the first input text and the first candidate output result.

[0144] In other words, when the first input text is non-compliant, the computer device can also perform correlation matching between each reference input text in the compliance database and each candidate output result to obtain each third correlation value; the higher the third correlation value, the higher the correlation between the first candidate output result and the first reference input text.

[0145] For example, the matching degree between the first input text and the first candidate output result can be the sum of the first relevance value, the second relevance value, and the third relevance value, or it can be the weighted average of the first relevance value, the second relevance value, and the third relevance value. The weight of the third relevance value can be preset by the developer.

[0146] Based on the above embodiments, this embodiment provides a feasible solution for obtaining the matching degree between the first input text and the first candidate output result. Specifically, it may include: based on the first correlation value and the second correlation value, the computer device obtains a third correlation value between the first candidate output result and the first reference input text; and calculates the matching degree by combining the first correlation value, the second correlation value, and the third correlation value. This solution also considers the matching between the reference input text in the compliance database and the candidate output result. On the one hand, due to the large amount of data in the compliance database, this solution can avoid the situation where the first candidate output result does not match the first input text due to errors in the compliance database; on the other hand, this solution can combine the correlation between the reference input text and the first input text to obtain the first candidate output result that best matches the first input text.

[0147] Based on the solutions shown in the above embodiments of this application, in one possible implementation, step 230b1 can be implemented as follows:

[0148] The first, second, and third relevance values ​​are weighted to obtain a weighted result; based on the weighted result, the matching degree between the first input text and the first candidate output result is obtained.

[0149] The aforementioned weighting process refers to adjusting the weights of the first, second, and third relevance values ​​(three relevance values) during calculation or analysis, reflecting their relative importance. Specifically, this weighting process can be weighted summation, weighted averaging, weighted distance / similarity calculation, or weighted statistic calculation. The computer can then output the weighted processing result as the matching degree between the first input text and the first candidate output result. Alternatively, the computer can perform predetermined processing (such as normalization) on the weighted processing result to obtain the matching degree between the first input text and the first candidate output result.

[0150] For example, developers can pre-set the weights corresponding to the first, second, and third relevance values ​​in their computer devices to form a weight set. Developers can then set different weight sets for each non-compliant type based on the non-compliant type of the first input text.

[0151] Based on the above embodiments, this embodiment illustrates how to obtain the matching degree between the first input text and the first candidate output result based on the first relevance value, the second relevance value, and the third relevance value. Specifically, it may include: a computer device performing weighted processing on the above three relevance values, and using the weighted result as the matching degree; the first relevance value, the second relevance value, and the third relevance value have different importance, and this solution can improve the accuracy and rationality of the matching degree calculation between the first input text and the first candidate output result.

[0152] For example, taking the first candidate output result as text, this application can obtain the first correlation value, the second correlation value, and the third correlation value through a multi-model dual-tower structure matching model.

[0153] Please refer to Figure 4 This illustrates a schematic diagram of the training and inference of a multi-model dual-tower structure matching model provided in an exemplary embodiment of this application. Figure 4 As shown, the multi-model dual-tower structure matching model includes feature encoder 1 and feature encoder 2.

[0154] The training process for feature encoder 1 is as follows:

[0155] Step A1: Input the first input text sample 401 and the first reference input text sample 402 into the feature encoder 1, wherein the correlation value 1 between the first input text sample 401 and the first reference input text sample 402 is known;

[0156] Step A2: Feature encoder 1 converts the first input text sample 401 and the first reference input text sample 402 into feature vectors respectively, and obtains the embedding (emd) features corresponding to the first input text sample 401 and the first reference input text sample 402 respectively.

[0157] Step A3: By comparing and learning the EMD features corresponding to the first input text sample 401 and the first reference input text sample 402 respectively, calculate the correlation value 2 between the first input text sample 401 and the first reference input text sample 402;

[0158] Step A4: Update feature encoder 1 based on the known correlation value 1 and the calculated correlation value 2.

[0159] The training process for feature encoder 2 is as follows:

[0160] Step B1: Input the first candidate output result sample 403, the first input text sample 401, and the first reference input text sample 402 into the feature encoder 2, wherein the correlation value 2 between the first candidate output result sample 403 and the first input text sample 401 and the correlation value 3 between the first candidate output result sample 403 and the first reference input text sample 402 are known;

[0161] Step B2: Feature encoder 2 converts the first input text sample 401, the first reference input text sample 402, and the first candidate output result sample 403 into feature vectors respectively, and obtains the EMD features corresponding to the first input text sample 401, the first reference input text sample 402, and the first candidate output result sample 403 respectively;

[0162] Step B3: By comparing and learning the EMD features corresponding to the first input text sample 401, the first reference input text sample 402, and the first candidate output result sample 403, calculate the correlation value 2 between the first candidate output result sample 403 and the first input text sample 401, and the correlation value 3 between the first candidate output result sample 403 and the first reference input text sample 402.

[0163] Step B4: Update feature encoder 2 based on the known correlation value 1 and the calculated correlation values ​​2 and 3.

[0164] During the inference process of the multi-model dual-tower structure matching model, the computer device inputs the first input text 404 and the first reference input text 405 into the feature encoder 1 to obtain the first correlation value 406 output by the feature encoder 1; at the same time, the computer device inputs the first candidate output result 407, the first input text 404, and the first reference input text 405 into the feature encoder 2 to obtain the second correlation value 408 and the third correlation value 409 output by the feature encoder 2.

[0165] Based on the solutions shown in the above embodiments, in some embodiments, the above... Figure 3 The text processing method shown further includes the following steps: when no matching output result exists in the compliance database, obtain a second candidate output result, which is the candidate output result with the highest matching degree between the first input text and the compliance database; the matching output result is the candidate output result whose matching degree with the first input text satisfies the matching condition; obtain a second descriptive text, which is used to describe the second candidate output result; generate a third input text based on the second descriptive text; pass the third input text to a generative language model that performs natural language processing to obtain a second output result returned by the generative language model; if the matching degree between the first input text and the second output result is greater than the matching degree threshold, determine the second output result as the first output result.

[0166] The statement "no matching output result exists in the compliance database" means that none of the candidate output results in the compliance database match the first input text. In this case, the computer device can obtain the second candidate output result corresponding to the highest matching value, and obtain the second description text of the second candidate output result according to the method for obtaining the first description text.

[0167] The above-mentioned generation of the third input text based on the second description text can be achieved by adding the second description text to the first input text to obtain the third input text; or by referring to the second description text, modifying the non-compliant description in the first input text, and changing the first input text into a compliant third input text.

[0168] The matching threshold can be a minimum matching value that the developer can pre-set on the computer device. When the matching degree between the first input text and the second output result is greater than the matching threshold, the computer device can regard the second output result as the first output result.

[0169] Based on the above embodiments, this embodiment illustrates a feasible solution for obtaining the first output result when the matching degree between the first input text and the first candidate output result does not meet the matching conditions. Specifically, it may include: the computer device can generate a third input text based on the candidate output result with the highest matching degree with the first input text in the compliance database; then, a second output result is obtained based on the third input text; when the matching degree between the first input text and the second output result meets the preset conditions, the second output result is determined as the first output result; this solution can ensure the compliance of the first output result while generating the first output result.

[0170] Based on the solutions shown in the above embodiments, in some embodiments, the above... Figure 3 The text processing method shown also includes the following steps: if no matching output result exists in the compliance database, modify the first input text to obtain a compliant fourth input text; the matching output result is a candidate output result whose matching degree with the first input text meets the matching condition; and pass the fourth input text to the generative language model that performs natural language processing to obtain the third output result returned by the generative language model.

[0171] In this context, "no matching output results exist in the compliance database" means that none of the candidate output results in the compliance database meet the matching criteria with the first input text. In this case, the computer device can modify the first input text to a compliant fourth input text.

[0172] For example, the above modification of the first input text to obtain a compliant fourth input text can be implemented as follows: the computer device obtains one or more guide words based on the first input text; then, the computer device adds one or more guide words to the first input text to obtain the fourth input text.

[0173] Please refer to Figure 5 This illustrates a flowchart of the process for obtaining guide words provided in an exemplary embodiment of this application. For example... Figure 5 As shown, the computer device analyzes and processes the first input text 501 to obtain the non-compliant type (one or more of type 1, 2...n) corresponding to the first input text 501; wherein, the developer user can pre-set one or more guide words corresponding to each non-compliant type in the computer device to establish a mapping set between non-compliant types and guide words; then, the computer device queries one or more guide words that have a corresponding relationship with the non-compliant type from the above mapping set according to the above non-compliant type.

[0174] For example, the aforementioned non-compliance types can indicate that the non-compliant first input text does not conform to one or more of the following: language specifications, content moderation, data privacy, laws and regulations, and ethical standards. For example, the mapping set between the aforementioned non-compliance types and guiding words can be in the form of a table as shown in Table 1. Specifically, when the first input text does not conform to language specifications, the computer device can select the corresponding guiding word based on the language of the first input text; for example, when the first input text does not conform to Chinese language specifications, the guiding word could be "requires conformity to Chinese language specifications".

[0175] Table 1

[0176]

[0177] For example, the non-compliant types of the first input text 501 include type 1 and type 2, which correspond to prompt word 1 and prompt word 2 respectively. The computer device can directly add prompt word 1 and prompt word 2 to the first input text; or, the computer device can also generate prompt text that conforms to language logic by simply supplementing prompt word 1 and prompt word 2 with simple text, and add the prompt text to the first input text 501.

[0178] Based on the above embodiments, this embodiment illustrates a feasible solution for obtaining the first output result when the matching degree between the first input text and the first candidate output result does not meet the matching conditions. Specifically, it may include: the computer device can modify the first input text into a compliant fourth input text; then, the computer device can pass the fourth input text to a generative language model, and after processing the fourth input text, the generative language model outputs a third output result corresponding to the fourth input text, which is the output result of the first input text; this solution can ensure the compliance of the first output result while generating the first output result.

[0179] Based on the solutions shown in the above embodiments, in some embodiments, when the compliance identification information indicates that the first input text is non-compliant, the compliance identification information is also used to indicate the risk level of the first input text; the above step 230 can be implemented as follows: when the risk level of the first input text is a specified risk level, obtain the first output result corresponding to the first input text based on the compliance database.

[0180] The aforementioned risk level can be the level of non-compliance of the first input text. In other words, the compliance identification information can indicate not only whether the first input text is compliant, but also the degree of non-compliance. For example, developers can predefine the standards for these risk levels in their computer devices. For instance, developers can determine the corresponding risk level classification standards based on relevant regulations, industry standards, corporate policies, and other application scenario specifications. These classification standards may include, but are not limited to: language standards, content moderation, data privacy, laws and regulations, ethical standards, and the probability, severity, and scope of impact of non-compliant information.

[0181] For example, the risk level of the aforementioned first input text can be divided into multiple levels (such as low risk, medium risk, high risk, extremely high risk, etc.). Developers can set specific thresholds or scoring rules for each level. For instance, first input text that does not conform to language standards can be classified as low risk; conversely, first input text that endangers public safety can be classified as extremely high risk.

[0182] In this embodiment, the specified risk level can be medium risk or high risk. That is, when the risk level of the first input text is medium risk or high risk, the computer device can obtain the first output result corresponding to the first input text based on the compliance database.

[0183] Optionally, if the risk level of the first input text is an unspecified risk level (such as low risk), the computer device transmits the first input text to a generative language model of natural language processing to obtain the output result returned by the generative language model.

[0184] In the above embodiments, this embodiment illustrates that compliance identification information can also be used to indicate the risk level of the first input text. This risk level can include multiple levels. When the risk level of the first input text is a specified risk level, the computer device can query the compliance database for candidate output results corresponding to the specified risk level and determine the candidate output result as the first output result of the first input text. For the first input text with a specified risk level, it is not necessary to follow the above... Figure 3 The proposed solution performs operations such as obtaining relevant information, and this solution can improve the processing efficiency of the first input text with a specified risk level.

[0185] Based on the solutions shown in the above embodiments, please refer to Figure 6 This illustrates a flowchart of a text processing method provided in an exemplary embodiment of this application. Figure 6 As shown above, Figure 2 Step 220 can be implemented as steps 220a and 220b.

[0186] Step 220a: Extract keywords from the first input text.

[0187] For example, keywords can be one or more word units that can represent the core content of the first input text; specifically, keywords can be multiple phrases that appear frequently in the first input text.

[0188] For example, keywords can be one or more word units in the first input text that are suspected of being non-compliant. Specifically, keywords can be the same as words / phrases / idioms in the compliance check thesaurus; or keywords can be highly similar to words / phrases / idioms in the non-compliance thesaurus.

[0189] The compliance check terminology can be a terminology pre-set by the developer user on the computer device. This compliance check terminology contains multiple non-compliant words / phrases / phrases that do not meet the requirements of language specifications, content review, data privacy, laws and regulations, ethical standards, etc.

[0190] Based on the solutions shown in the above embodiments of this application, in one possible implementation, step 220a can be implemented as follows:

[0191] Perform word segmentation on the first input text to obtain each candidate word in the first input text;

[0192] Among the candidate words, those existing in the compliance check terminology library are extracted as keywords; the compliance check terminology library contains words that pose a risk of non-compliance.

[0193] The computer device can select a word segmentation tool to segment the first input text according to the language of the first input text (such as Chinese or English), input the first input text into the word segmentation tool, and obtain multiple candidate words output by the word segmentation tool.

[0194] Next, the candidate words are filtered, and the candidate words that are the same as the non-compliant words in the compliance check terminology are retained. The candidate words are then determined as the keywords of the first input text.

[0195] For example, the aforementioned compliance check terminology can be a terminology pre-set by the developer user on their computer device. For instance, the developer user collects several non-compliant terms and expands them with semantically similar terms; the non-compliant terms and expanded terms together constitute the aforementioned compliance check terminology.

[0196] Based on the above embodiments, this embodiment shows how to extract keywords from the first input text, which may specifically include: obtaining multiple candidate words in the first input text through word segmentation processing, and determining the part or all of the candidate words that belong to non-compliance as keywords; this solution can improve the recognition frequency of non-compliant words, avoid missed recognition, and improve the accuracy of compliance recognition information.

[0197] Exemplarily, the above step 220a can also be implemented as:

[0198] 1) Word segmentation: Remove one or more of the following from the first input text: punctuation marks, special characters, numbers, stop words (such as common but insignificant words like "of", "is", "in", etc.); then, split the remaining content of the first input text into separate lexical units.

[0199] 2) Importance measurement: A pre-trained word vector model can be used to calculate the word vectors of each lexical unit, and then through measurement methods such as cosine similarity and Euclidean distance, find the lexical units that are closer to other word vectors, and determine these lexical units as the keywords of the first input text.

[0200] 3) Filtering: Calculate the semantic similarity between each lexical unit and the non-compliant words in the compliance check library, and also determine the lexical units whose semantic similarity with the non-compliant words meets the preset conditions as the keywords of the first input text; among them, the developer user can preset the above preset conditions in the computer device. Specifically, for example, map each lexical unit and each non-compliant word to word vectors respectively, and calculate the semantic similarity between the lexical unit and the non-compliant word through the cosine similarity of the word vectors. The above preset condition can be that the cosine similarity is greater than 0.8.

[0201] Step 220b: Input the first input text and the keywords into the compliance recognition model, and obtain the compliance recognition information output by the compliance recognition model.

[0202] Among them, the compliance recognition model is a machine learning model trained based on input text samples and annotation information; the annotation information is used to indicate whether the input text sample is compliant.

[0203] During the training process of the compliance recognition model, input the input text sample and the annotation information of the input text sample into the compliance recognition model, and obtain the compliance recognition information output by the compliance recognition model; then, compare the compliance recognition information with the annotation information and update the parameters of the compliance recognition model.

[0204] During the inference process of the compliance recognition model, input the first input text and the keywords into the compliance recognition model, and obtain the compliance recognition information output by the compliance recognition model.

[0205] For example, the compliance identification model described above may include a Perceiver model and a classifier.

[0206] like Figure 7 As shown, it illustrates a training and inference diagram of a compliance identification model provided in an exemplary embodiment of this application. Figure 7 As shown, during training, the latent space features and the input text sample 701 calculate cross-attention, interacting through Cross-Attention. This reduces the feature dimension of the latent space to a very small value, with a computational cost of O(MN) + O(M^2), thus improving computational speed. Furthermore, the Perceiver model can handle input text samples 701 of variable length. During inference, after calculating cross-attention with the first input text 702, the latent space features also need to calculate cross-attention with the keyword (tag) 703.

[0207] Based on the above embodiments, this embodiment provides a feasible solution for obtaining compliance identification information of the first input text, which may specifically include: extracting keywords from the first input text, using the keywords and the first input text as input to the compliance identification model, and obtaining compliance identification information of the first input text; this solution can increase the proportion of keywords in the first input text in the compliance identification process and improve the accuracy of compliance identification information.

[0208] Based on the above embodiments, please refer to Figure 8 This illustrates a flowchart of a text processing method provided in an exemplary embodiment of this application. Figure 8 As shown, firstly, the computer device can input the acquired first input text 801 into the compliance recognition model. The compliance recognition model can perform compliance recognition processing on the first input text 801 and output the compliance recognition information 802 of the first input text 801. Optionally, the computer device can extract suspected non-compliant words from the first input text 801, input the first input text 801 and the suspected non-compliant words into the compliance recognition model, and obtain the compliance recognition information output by the compliance recognition model.

[0209] Secondly, the compliance identification information 802 can indicate whether the first input text 801 is compliant. When the compliance identification information 802 indicates that the first input text 801 is compliant, the computer device can input the first input text 801 into the generative language model 8a, which can perform natural language processing on the first input text 801 and output a first output result 803. When the compliance identification information 802 indicates that the first input text 801 is non-compliant, the computer device can retrieve a first candidate output result 804 corresponding to the first input text 801 from the compliance database 8b.

[0210] Then, the computer device can determine whether the matching degree between the first input text 801 and the first candidate output result 804 meets the matching conditions. Specifically, this includes the following two cases:

[0211] Scenario 1: When the matching degree between the first input text 801 and the first candidate output result 804 meets the matching condition, the computer device can obtain the first output result 805 based on the first candidate output result 804. For example, the computer device can determine the first candidate output result 804 as the first output result 805; or, the computer device can also modify the non-compliant first input text 801 into a compliant second input text 806 based on the first candidate output result 804, and then input the second input text 806 into the generative language model 8a.

[0212] Case 2: When the matching degree between the first input text 801 and the first candidate output result 804 does not meet the matching condition, the computer device can obtain the compliant third input text 807 based on the candidate output result with the highest matching degree between the first input text 801 and the compliant database 8b, and then input the third input text 807 into the generative language model 8a.

[0213] In addition, the compliance identification information 802 can also indicate the risk level of the first input text 801. When the risk level of the first input text 801 reaches a specified risk level (such as a medium risk level), the computer device can obtain the first candidate output result 804 corresponding to the first input text 801 from the compliance database 8b; then, the computer device can execute the steps shown in Case 1 and Case 2 above according to whether the matching degree between the first input text 801 and the first candidate output result 804 meets the matching conditions, which will not be repeated here.

[0214] Optionally, when the risk level of the first input text 801 is low, the computer device can transmit the first input text 801 to the generative language model 8a to obtain the first output result 803 output by the generative language model 8a.

[0215] On the one hand, the text processing method shown in the above embodiments of this application can be applied to AIGC algorithm scenarios such as text-to-text, text-to-image, text-to-video, and text-to-3D. For example, the above text processing method can be applied to the text answering scenario of a large language model (i.e., text-to-text), wherein the first input text is the first prompt, the compliance database is the compliance question-and-answer database, the candidate output result is the candidate answer text, the reference input text is the reference prompt, the generative language model is the large language model, the second input text is the second prompt, and the third input text is the third prompt.

[0216] Please refer to Figure 9 This illustrates a flowchart of a prompt word processing method provided in an exemplary embodiment of this application. Figure 9 As shown, the text response processing method includes the following steps:

[0217] Step 910: Obtain the first prompt word, which is the input text for natural language processing;

[0218] Step 920: Obtain the compliance identification information of the first prompt word. The compliance identification information is used to indicate whether the first prompt word is compliant.

[0219] Step 930: If the compliance identification information indicates that the first prompt word is non-compliant, obtain the first answer text corresponding to the first prompt word based on the compliance question and answer database; the compliance question and answer database stores compliant candidate answer texts.

[0220] Optionally, the candidate answer texts in the compliance question-and-answer database have corresponding reference prompts;

[0221] Step 930 above can be implemented as follows:

[0222] Step 930a: If the compliance identification information indicates that the first prompt word is non-compliant, obtain relevant information, including a first relevance value and a second relevance value; the first relevance value is used to indicate the relevance between the first prompt word and the first reference prompt word, and the second relevance value is used to indicate the relevance between the first prompt word and the first candidate answer text; the first candidate answer text is a candidate answer text in the compliance question and answer database, and the first candidate answer text corresponds to the first reference prompt word;

[0223] Step 930b: Based on relevant information, obtain the matching degree between the first prompt word and the first candidate answer text;

[0224] Step 930c: If the matching degree between the first prompt word and the first candidate answer text meets the matching condition, obtain the first answer text based on the first candidate answer text.

[0225] Optionally, step 930c above can be implemented as follows:

[0226] Step 930c1: Based on the first candidate answer text, modify the first prompt word to obtain the second prompt word;

[0227] Step 930c2: Pass the second prompt word to the large language model that performs natural language processing, and obtain the first answer text returned by the large language model.

[0228] Optionally, step 930c1 above can be implemented as follows:

[0229] Obtain the first description text, which is used to describe the first candidate answer text;

[0230] Add the first descriptive text to the first prompt word to obtain the second prompt word.

[0231] Optionally, the relevant information also includes a third relevance value, which is used to indicate the relevance between the first candidate answer text and the first reference prompt word;

[0232] The above step 930b can be implemented as follows:

[0233] Step 930b1: Based on the first relevance value, the second relevance value, and the third relevance value, obtain the matching degree between the first prompt word and the first candidate answer text.

[0234] Optionally, step 930b1 above can be implemented as follows:

[0235] The first, second, and third correlation values ​​are weighted to obtain the weighted result.

[0236] Based on the weighted processing results, the matching degree between the first prompt word and the first candidate answer text is obtained.

[0237] Optionally, the above method further includes:

[0238] If no matching answer text exists in the compliance Q&A database, a second candidate answer text is obtained. The second candidate answer text is the candidate answer text with the highest matching degree with the first prompt word in the compliance Q&A database. The matching answer text is the candidate answer text whose matching degree with the first prompt word meets the matching condition.

[0239] Obtain the second description text, which is used to describe the second candidate answer text;

[0240] Generate a third prompt word based on the second descriptive text;

[0241] The third prompt word is passed to a large language model that performs natural language processing to obtain the second answer text returned by the large language model.

[0242] If the match between the first prompt word and the second response text is greater than the match threshold, the second response text is identified as the first response text.

[0243] Optionally, the above method further includes:

[0244] If no matching answer text exists in the compliant Q&A database, modify the first prompt word to obtain a compliant fourth prompt word; the matching answer text is the candidate answer text whose matching degree with the first prompt word meets the matching condition;

[0245] The fourth prompt word is passed to the large language model that performs natural language processing, and the third response text returned by the large language model is obtained.

[0246] Optionally, if the compliance identification information indicates that the first prompt word is non-compliant, the compliance identification information may also be used to indicate the risk level of the first prompt word;

[0247] Step 930 above can be implemented as follows:

[0248] If the risk level of the first prompt word is a specified risk level, the first answer text corresponding to the first prompt word is obtained based on the compliance Q&A database.

[0249] Optionally, step 920 above can be implemented as follows:

[0250] Step 920a: Extract keywords from the first prompt word;

[0251] Step 920b: Input the first prompt word and keywords into the compliance identification model to obtain the compliance identification information output by the compliance identification model;

[0252] The compliance identification model is a machine learning model trained based on prompt word samples and annotation information; the annotation information is used to indicate whether the prompt word samples are compliant.

[0253] Optionally, step 920a above can be implemented as follows:

[0254] Perform word segmentation on the first prompt word to obtain each candidate word in the first prompt word;

[0255] Among the candidate words, those existing in the compliance check terminology library are extracted as keywords; the compliance check terminology library contains words that pose a risk of non-compliance.

[0256] For example, based on Figure 2 , Figure 3 , Figure 6 as well as Figure 9 For any one or more embodiments, this application proposes a method for optimizing the content ecology security of a large language model. This method can accurately identify the existing ecological risks in the dialogue of a large language model and guide the large language model to avoid the dangers, effectively reducing the leakage of ecological harm risks and improving user experience.

[0257] Please refer to Figure 10 This illustrates a flowchart of a method for optimizing the content ecosystem security of a large language model provided in an exemplary embodiment of this application. Figure 10As shown, firstly, the computer device obtains the input content 1001 queried by the user from the large language model; secondly, the input content 1001 is transmitted to the ecological risk identification model 1002 (i.e. the compliance identification model mentioned above), and ecological risk identification processing is performed on the user's input content 1001 to obtain the risk level corresponding to the input content 1001; the risk level can be divided into three levels: level one risk, level two risk, and level three risk.

[0258] Among them, Level 1 risk corresponds to the lowest risk level, that is, the input content 1001 corresponding to Level 1 risk is compliant. For the input content 1001 corresponding to Level 1 risk, the computer device generates the corresponding direct response content 1003 based on the input content 1001 through the large language model.

[0259] For input content 1001 corresponding to Level 2 risk (i.e., the aforementioned specified risk level), on one hand, the computer device can retrieve positive compliance answers from the white question database 1005 (i.e., the aforementioned compliance question-and-answer database) and use the candidate output results 1006 from the white question database 1005 as the output content of the large language model. On the other hand, the computer device can also process the input content 1001 with adaptive prompt words 1007 (i.e., add the aforementioned prompt words to the input content 1001). Figure 5 The guiding words (as shown) guide the large language model to generate positive and compliant response content 1008. The compliant response content 1008 generated by the large language model can also be passed to the ecological risk identification model 1002 for the aforementioned ecological risk identification processing to verify whether the compliant response content 1008 meets the risk requirements. For example, to improve the processing efficiency of the input content 1001 corresponding to the secondary risk, the computer device prioritizes searching for positive and compliant response content in the white question database 1005.

[0260] Among them, Level 3 risk corresponds to the highest risk level, meaning that input content 1001 is non-compliant. For input content 1001 corresponding to Level 3 risk, the computer device can output a predetermined output result 1004. For example, when input content 1001 involves information that endangers public safety and interests, the predetermined output result 1004 could be "The server encountered a problem, please wait." Furthermore, the computer device can take necessary intervention measures based on the specific non-compliant information in input content 1001, such as deleting text, freezing accounts, and notifying relevant agencies. Specifically, if the first input text involves the user's life safety, the computer device can activate alarm measures.

[0261] Alternatively, the ecological risk identification model 1002 can be Figure 7The compliance identification model shown. During the training of the Perceiver model, rules need to be defined and labeled for the input text samples. This application can divide the training of the ecological security model into multiple batches, and through iterative learning, select data with labeling value for annotation. For example... Figure 11 As shown, the computer device transmits the input text sample 1101 containing data annotations to the compliance recognition model 1102 for training, which can obtain the difficult-to-classify samples 1103 in the input text sample 1101. Then, the computer device can use a decision boundary sampling method to filter the difficult-to-classify samples 1103, selecting positive samples from the difficult-to-classify samples 1103. After annotating these positive samples, they are added to the input text sample 1101, allowing the positive samples in the difficult-to-classify samples 1103 to be used for training, thereby accelerating the learning efficiency of the Perceiver model.

[0262] like Figure 12 As shown, the computer device divides the input text sample 1101 into positive samples 1201, negative samples 1202, and difficult-to-classify samples 1204 through a decision boundary 1203. Positive samples 1201 can be non-compliant information that meets the conditions, while negative samples 1202 can be compliant information. In ecological security scenarios, due to the small proportion of positive samples, indiscriminate labeling of the input text samples is not only inefficient but also leads to poor learning performance of the Perceiver model due to insufficient positive samples. Therefore, this application can perform data filtering on the difficult-to-classify samples 1204 near the decision boundary 1203, selecting positive samples from the difficult-to-classify samples 1204 and adding them to the training samples of the Perceiver model.

[0263] In the process of retrieving positive answers from the white question database, the retrieval algorithm in the relevant technology adopts a dual-tower structure and shares an encoder, aligning the query and answer through contrastive learning. Since the distribution of the user-input prompt and the designed white question database differs to some extent, directly using the dual-tower model of the white question database for user input can easily lead to mismatches when the number of white questions is large.

[0264] Optionally, in order to reduce the discrepancy between user input and the white-hat database and improve the relevance of search results, this application adopts... Figure 4 The multi-model dual-tower retrieval model is shown. Figure 4The first input text 404 is the user input (query1), the first reference input text 405 is the white question (query2), and the first candidate output result 407 is the answer (answer). This application also designs a new dual-tower structure for the query domain, which not only considers the correlation between the user input (query1) and the white question (query2), but also the correlation between the user input (query1) and the answer (answer), greatly improving the matching accuracy of the white question database and the user experience.

[0265] Optionally, in the application scenario of large language models, computer devices can identify ecological risks in the text content of a Prompt after the user inputs it, design guiding words for different types of violations, and provide positive guidance to the large language model. For long-tailed sparse risky content, this application improves the retrieval algorithm, designs a whitelist of questions, and performs high-precision retrieval on it to ensure that the generated text complies with language norms, content review, data privacy, laws and regulations, and ethical standards. For example, if the user's input Prompt contains content that does not comply with laws and regulations, the ecological security identification model can identify that the Prompt is non-compliant; then, based on the risk level of the ecological risks present in the Prompt, the security of the final result can be ensured by adding prompt words and performing high-precision retrieval from the whitelist of questions. Compared with refusing to answer, this soft interception scheme can improve the user experience.

[0266] In summary, the method described in the above embodiments of this application can be applied to a high-precision large language model ecosystem security optimization scheme, applicable to AIGC tasks of various modalities, including text-to-text, text-to-image, text-to-video, and text-to-3D. This scheme employs the Perceiver model, which supports longer user input text, resulting in higher model accuracy and more accurate ecosystem risk identification. This application proposes an active learning iteration scheme and an adaptive guide word soft interception scheme, increasing positive sample annotation efficiency, reducing labor costs, and improving user experience and interactivity. This application improves the retrieval algorithm of the dual-tower model, enhancing the matching accuracy between user prompts and the white question database, further reducing security risks related to the large language model ecosystem.

[0267] From a practical perspective, this application proposes an ecological security identification algorithm that supports long texts and designs an active learning algorithm iteration scheme to significantly improve the efficiency of manual annotation of positive samples. When ecological problems are detected in the input or output text, on the one hand, by leveraging the powerful linguistic logic capabilities of the large language model, corresponding prompt words are designed to guide the model to avoid ecological risks. This soft interception technique provides users with a better user experience. On the other hand, this method also designs a dual-tower structure retrieval algorithm with multiple models, which greatly improves the retrieval accuracy of the white question database and provides positive answers to long-tail questions that the large language model is suspected of causing harm, reducing ecological problems that may arise during the use of the large language model.

[0268] On the other hand, the method shown in the above embodiments of this application can be applied to query scenarios in search systems. For example, by applying the above text processing method to the underage mode scenario of a search system, in order to protect minors, the search system can perform hierarchical protection processing on the query results of the search system in scenarios where minors use the search system. Here, the first input text is the first query text, the compliance database is the hierarchical question-and-answer database, the candidate output results are candidate query results, the reference input text is the reference query text, the generative language model is the query search system, the second input text is the second query text, and the third input text is the third query text.

[0269] Please refer to Figure 13 This illustrates a flowchart of a query text processing method provided in an exemplary embodiment of this application. Figure 13 Therefore, the query text processing method includes the following steps:

[0270] Step 1310: Obtain the first query text, which is the input text for natural language processing;

[0271] Step 1320: Obtain the compliance identification information of the first query text. The compliance identification information is used to indicate whether the first query text is compliant.

[0272] Step 1330: If the compliance identification information indicates that the first query text is non-compliant, obtain the first query result corresponding to the first query text based on the hierarchical question and answer library; the hierarchical question and answer library stores compliant candidate query results.

[0273] Optionally, the candidate query results in the hierarchical question-and-answer database correspond to reference query text;

[0274] Step 1330 above can be implemented as follows:

[0275] Step 1330a: If the compliance identification information indicates that the first query text is non-compliant, obtain relevant information, including a first relevance value and a second relevance value; the first relevance value is used to indicate the relevance between the first query text and the first reference query text, and the second relevance value is used to indicate the relevance between the first query text and the first candidate query result; the first candidate query result is a candidate query result in the hierarchical question-answering library, and the first candidate query result corresponds to the first reference query text;

[0276] Step 1330b: Based on relevant information, obtain the matching degree between the first query text and the first candidate query result;

[0277] Step 1330c: If the matching degree between the first query text and the first candidate query result meets the matching condition, obtain the first query result based on the first candidate query result.

[0278] Optionally, step 1330c above can be implemented as follows:

[0279] Step 1330c1: Based on the first candidate query result, modify the first query text to obtain the second query text;

[0280] Step 1330c2: Pass the second query text to the query search system that performs natural language processing, and obtain the first query result returned by the query search system.

[0281] Optionally, step 1330c1 above can be implemented as follows:

[0282] Obtain the first description text, which is used to describe the first candidate query result;

[0283] Add the first description text to the first query text to obtain the second query text.

[0284] Optionally, the relevant information also includes a third relevance value, which is used to indicate the relevance between the first candidate query result and the first reference query text;

[0285] Step 1330b above can be implemented as follows:

[0286] Step 1330b1: Based on the first relevance value, the second relevance value, and the third relevance value, obtain the matching degree between the first query text and the first candidate query result.

[0287] Optionally, step 1330b1 above can be implemented as follows:

[0288] The first, second, and third correlation values ​​are weighted to obtain the weighted result.

[0289] Based on the weighted processing results, the matching degree between the first query text and the first candidate query result is obtained.

[0290] Optionally, the above method further includes:

[0291] If no matching query result exists in the hierarchical question-and-answer database, a second candidate query result is obtained. The second candidate query result is the candidate query result with the highest matching degree with the first query text in the hierarchical question-and-answer database. The matching query result is the candidate query result whose matching degree with the first query text meets the matching condition.

[0292] Obtain the second description text, which is used to describe the second candidate query result;

[0293] Generate the third query text based on the second description text;

[0294] The third query text is passed to the query search system that performs natural language processing to obtain the second query result returned by the query search system.

[0295] If the matching degree between the first query text and the second query result is greater than the matching degree threshold, the second query result will be determined as the first query result.

[0296] Optionally, the above method further includes:

[0297] If no matching query result exists in the hierarchical question-and-answer database, modify the first query text to obtain a compliant fourth query text; the matching query result is the candidate query result whose matching degree with the first query text meets the matching condition.

[0298] The fourth query text is passed to the query search system that performs natural language processing, and the third query result returned by the query search system is obtained.

[0299] Optionally, if the compliance identification information indicates that the first query text is non-compliant, the compliance identification information may also be used to indicate the risk level of the first query text;

[0300] Step 1330 above can be implemented as follows:

[0301] If the risk level of the first query text is a specified risk level, the first query result corresponding to the first query text is obtained based on the hierarchical question and answer database.

[0302] Optionally, step 1320 above can be implemented as follows:

[0303] Step 1320a: Extract keywords from the first query text;

[0304] Step 1320b: Input the first query text and keywords into the compliance identification model to obtain the compliance identification information output by the compliance identification model;

[0305] The compliance identification model is a machine learning model trained based on query text samples and annotation information; the annotation information is used to indicate whether the query text samples are compliant.

[0306] Optionally, step 1320a above can be implemented as follows:

[0307] Perform word segmentation on the first query text to obtain each candidate word in the first query text;

[0308] Among the candidate words, those existing in the compliance check terminology library are extracted as keywords; the compliance check terminology library contains words that pose a risk of non-compliance.

[0309] Please refer to Figure 14 The diagram illustrates a block diagram of a text processing apparatus according to an exemplary embodiment of this application, which can be used to perform tasks such as... Figure 2 , Figure 3 or Figure 6 In the method shown, all or part of the steps performed by the computer device are as follows: Figure 14 As shown, the device includes:

[0310] The input acquisition module 1401 is used to acquire the first input text, which is the input text for natural language processing.

[0311] The information acquisition module 1402 is used to acquire compliance identification information of the first input text, and the compliance identification information is used to indicate whether the first input text is compliant;

[0312] The output acquisition module 1403 is used to acquire the first output result corresponding to the first input text based on the compliance database when the compliance identification information indicates that the first input text is non-compliant; the compliance database stores compliant candidate output results.

[0313] In some embodiments, the candidate output results in the compliance database correspond to reference input text; the output acquisition module 1403 is used for:

[0314] If the compliance identification information indicates that the first input text is non-compliant, relevant information is obtained, including a first correlation value and a second correlation value. The first correlation value is used to indicate the correlation between the first input text and the first reference input text, and the second correlation value is used to indicate the correlation between the first input text and the first candidate output result. The first candidate output result is a candidate output result in the database, and the first candidate output result corresponds to the first reference input text.

[0315] Based on relevant information, obtain the matching degree between the first input text and the first candidate output result;

[0316] If the matching degree between the first input text and the first candidate output result meets the matching condition, the first output result is obtained based on the first candidate output result.

[0317] In some embodiments, the output acquisition module 1403 is used for:

[0318] Based on the first candidate output result, modify the first input text to obtain the second input text;

[0319] The second input text is passed to the generative language model that performs natural language processing, and the first output result returned by the generative language model is obtained.

[0320] In some embodiments, the output acquisition module 1403 is used for:

[0321] Obtain the first description text, which is used to describe the first candidate output result;

[0322] Add the first descriptive text to the first input text to obtain the second input text.

[0323] In some embodiments, the relevant information also includes a third correlation value, which is used to indicate the correlation between the first candidate output and the first reference input text;

[0324] The output acquisition module 1403 is used to obtain the matching degree between the first input text and the first candidate output result based on the first relevance value, the second relevance value and the third relevance value.

[0325] In some embodiments, the output acquisition module 1403 is used for:

[0326] The first, second, and third correlation values ​​are weighted to obtain the weighted result.

[0327] Based on the weighted processing result, the matching degree between the first input text and the first candidate output result is obtained.

[0328] In some embodiments, the apparatus further includes: a text generation module, configured to:

[0329] If no matching output result exists in the compliance database, a second candidate output result is obtained. The second candidate output result is the candidate output result with the highest matching degree with the first input text in the compliance database. The matching output result is the candidate output result whose matching degree with the first input text satisfies the matching condition.

[0330] Obtain the second description text, which is used to describe the second candidate output result;

[0331] Generate the third input text based on the second description text;

[0332] The third input text is passed to the generative language model that performs natural language processing, and the second output result returned by the generative language model is obtained.

[0333] If the matching degree between the first input text and the second output result is greater than the matching degree threshold, the second output result is determined as the first output result.

[0334] In some embodiments, the apparatus further includes a text editing module, configured to:

[0335] If no matching output result exists in the compliance database, modify the first input text to obtain a compliant fourth input text; the matching output result is the candidate output result whose matching degree with the first input text meets the matching condition;

[0336] The fourth input text is passed to the generative language model that performs natural language processing, and the third output result is obtained from the generative language model.

[0337] In some embodiments, where compliance identification information indicates that the first input text is non-compliant, the compliance identification information is also used to indicate the risk level of the first input text;

[0338] The output acquisition module 1403 is used to acquire the first output result corresponding to the first input text based on the compliance database when the risk level of the first input text is a specified risk level.

[0339] In some embodiments, the information acquisition module 1402 is used for:

[0340] Extract keywords from the first input text;

[0341] Input the first input text and keywords into the compliance identification model to obtain the compliance identification information output by the compliance identification model;

[0342] The compliance identification model is a machine learning model trained based on input text samples and annotation information; the annotation information is used to indicate whether the input text sample is compliant.

[0343] In some embodiments, the information acquisition module 1402 is used for:

[0344] Perform word segmentation on the first input text to obtain each candidate word in the first input text;

[0345] Among the candidate words, those existing in the compliance check terminology library are extracted as keywords; the compliance check terminology library contains words that pose a risk of non-compliance.

[0346] It should be noted that the device provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules according to actual needs, that is, the content structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0347] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments of the relevant method; the technical effects achieved by each module performing its operation are the same as the technical effects in the embodiments of the relevant method, and will not be elaborated here.

[0348] Please refer to Figure 15 This diagram illustrates a structural block diagram of a computer device 1500 according to an exemplary embodiment of this application. The computer device can be implemented as a server as described above in this application. The computer device 1500 includes a Central Processing Unit (CPU) 1501, a system memory 1504 including Random Access Memory (RAM) 1502 and Read-Only Memory (ROM) 1503, and a system bus 1505 connecting the system memory 1504 and the CPU 1501. The computer device 1500 also includes a mass storage device 1506 for storing an operating system 1509, application programs 1510, and other program modules 1511.

[0349] The mass storage device 1506 is connected to the central processing unit 1501 via a mass storage controller (not shown) connected to the system bus 1505. The mass storage device 1506 and its associated computer-readable media provide non-volatile storage for the computer device 1500. That is, the mass storage device 1506 may include computer-readable media (not shown) such as a hard disk or a compact disc read-only memory (CD-ROM) drive.

[0350] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state storage technologies, CD-ROM, digital versatile disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer storage medium is not limited to the above-mentioned types. The system memory 1504 and mass storage device 1506 described above can be collectively referred to as memory.

[0351] According to various embodiments of this disclosure, the computer device 1500 can also be connected to a remote computer on a network, such as the Internet. That is, the computer device 1500 can be connected to a network 1508 via a network interface unit 1507 connected to the system bus 1505, or the network interface unit 1507 can be used to connect to other types of networks or remote computer systems (not shown).

[0352] The memory also includes at least one computer instruction stored in the memory, and the central processing unit 1501 executes the at least one computer instruction to implement all or part of the steps in the methods shown in the above embodiments.

[0353] In an exemplary embodiment, a chip is also provided, the chip including programmable logic circuits and program instructions, which, when the chip is run on a computer device, are used to implement the text processing method described above.

[0354] In an exemplary embodiment, a computer program product is also provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions to implement the text processing methods provided in the above-described method embodiments.

[0355] In an exemplary embodiment, a computer-readable storage medium is also provided, which stores computer instructions that are loaded and executed by a processor to implement the text processing methods provided in the above-described method embodiments.

[0356] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0357] Those skilled in the art will recognize that the functions described in the embodiments of this application in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of a computer program from one place to another. Storage media can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0358] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A text processing method, characterized in that, The method includes: Obtain the first input text, which is the input text for natural language processing; Obtain compliance identification information of the first input text, wherein the compliance identification information is used to indicate whether the first input text is compliant; If the compliance identification information indicates that the first input text is non-compliant, a first output result corresponding to the first input text is obtained based on the compliance database; the compliance database stores compliant candidate output results.

2. The method according to claim 1, characterized in that, The candidate output results in the compliance database correspond to reference input text; When the compliance identification information indicates that the first input text is non-compliant, the step of obtaining a first output result corresponding to the first input text based on the compliance database includes: When the compliance identification information indicates that the first input text is non-compliant, relevant information is obtained, including a first correlation value and a second correlation value; the first correlation value is used to indicate the correlation between the first input text and the first reference input text, and the second correlation value is used to indicate the correlation between the first input text and the first candidate output result; the first candidate output result is a candidate output result in the compliance database, and the first candidate output result corresponds to the first reference input text; Based on the aforementioned relevant information, the matching degree between the first input text and the first candidate output result is obtained; If the matching degree between the first input text and the first candidate output result meets the matching condition, the first output result is obtained based on the first candidate output result.

3. The method according to claim 2, characterized in that, The step of obtaining the first output result based on the first candidate output result when the matching degree between the first input text and the first candidate output result meets the matching condition includes: Based on the first candidate output result, modify the first input text to obtain the second input text; The second input text is passed to the generative language model that performs the natural language processing, and the first output result returned by the generative language model is obtained.

4. The method according to claim 3, characterized in that, The step of modifying the first input text based on the first candidate output result to obtain the second input text includes: Obtain a first description text, which is used to describe the first candidate output result; Add the first description text to the first input text to obtain the second input text.

5. The method according to claim 2, characterized in that, The relevant information also includes a third correlation value, which is used to indicate the correlation between the first candidate output result and the first reference input text; The step of obtaining the matching degree between the first input text and the first candidate output result based on the relevant information includes: Based on the first correlation value, the second correlation value, and the third correlation value, the matching degree between the first input text and the first candidate output result is obtained.

6. The method according to claim 5, characterized in that, The step of obtaining the matching degree between the first input text and the first candidate output result based on the first correlation value, the second correlation value, and the third correlation value includes: The first correlation value, the second correlation value, and the third correlation value are weighted to obtain a weighted processing result. Based on the weighted processing result, the matching degree between the first input text and the first candidate output result is obtained.

7. The method according to claim 2, characterized in that, The method further includes: If no matching output result exists in the compliance database, a second candidate output result is obtained. The second candidate output result is the candidate output result with the highest matching degree between the first input text and the compliance database. The matching output result is the candidate output result whose matching degree with the first input text satisfies the matching condition. Obtain a second description text, which is used to describe the second candidate output result; Based on the second description text, generate the third input text; The third input text is passed to the generative language model that performs the natural language processing to obtain the second output result returned by the generative language model; If the matching degree between the first input text and the second output result is greater than the matching degree threshold, the second output result is determined as the first output result.

8. The method according to claim 2, characterized in that, The method further includes: If no matching output result exists in the compliance database, the first input text is modified to obtain a compliant fourth input text; the matching output result is a candidate output result whose matching degree with the first input text satisfies the matching condition. The fourth input text is passed to the generative language model that performs the natural language processing to obtain the third output result returned by the generative language model.

9. The method according to any one of claims 1 to 8, characterized in that, If the compliance identification information indicates that the first input text is non-compliant, the compliance identification information is also used to indicate the risk level of the first input text; When the compliance identification information indicates that the first input text is non-compliant, the step of obtaining a first output result corresponding to the first input text based on the compliance database includes: If the risk level of the first input text is a specified risk level, the first output result corresponding to the first input text is obtained based on the compliance database.

10. The method according to any one of claims 1 to 8, characterized in that, The step of obtaining compliance identification information for the first input text includes: Extract keywords from the first input text; Input the first input text and the keywords into the compliance identification model to obtain the compliance identification information output by the compliance identification model; The compliance identification model is a machine learning model trained based on input text samples and annotation information; the annotation information is used to indicate whether the input text sample is compliant.

11. The method according to claim 10, characterized in that, The extraction of keywords from the first input text includes: Perform word segmentation on the first input text to obtain each candidate word in the first input text; The candidate words that exist in the compliance check terminology library are extracted as the keywords; the compliance check terminology library contains words that have the risk of non-compliance.

12. A text processing device, characterized in that, The device includes: The input acquisition module is used to acquire the first input text, which is the input text for natural language processing. The information acquisition module is used to acquire compliance identification information of the first input text, wherein the compliance identification information is used to indicate whether the first input text is compliant; The output acquisition module is used to acquire a first output result corresponding to the first input text based on a compliance database when the compliance identification information indicates that the first input text is non-compliant; the compliance database stores compliant candidate output results.

13. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one computer instruction, which is loaded and executed by the processor to implement the text processing method as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer instruction, which is loaded and executed by a processor to implement the text processing method as described in any one of claims 1 to 11.

15. A computer program product, characterized in that, The computer program product includes computer instructions stored in a computer-readable storage medium; the computer instructions are read and executed by a processor of a computer device to implement the text processing method as described in any one of claims 1 to 11.

Images