Method and device for detecting output text of large model, equipment, medium and program product

Abstract

The invention provides a method, a device, equipment, a medium and a program product for detecting an output text of a large model, which can be applied to the technical field of artificial intelligence and the technical field of financial science and technology. The method comprises the steps that a to-be-tested text output by a large model is obtained, and the large model executes a generative task according to input information of a user to obtain the to-be-tested text; matching the to-be-tested text with a preset dictionary tree to obtain a matching result, a plurality of nodes of the preset dictionary tree representing a plurality of preset characters related to a preset abnormal word, a side relationship among the plurality of nodes representing a position relationship with a plurality of preset characters in a preset character string, and the preset character string representing the preset abnormal word; and according to a matching result, determining abnormal words with semantic violation risks from the to-be-tested text.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence technology and financial technology technology, and more specifically to a method, apparatus, device, medium and program product for detecting the output text of a large model. Background Technology

[0002] Large language models, with their powerful natural language understanding and generation capabilities, have demonstrated enormous application potential in numerous fields. Before deploying a large language model, it is necessary to test it to avoid any abnormal results.

[0003] In related technologies, testing large language models requires excessive human resources, often resulting in low testing efficiency when faced with massive amounts of data, and difficulty in improving testing accuracy. Summary of the Invention

[0004] In view of the above problems, this application provides a method, apparatus, device, medium and program product for detecting the output text of a large model.

[0005] According to a first aspect of this application, a method for detecting the output text of a large model is provided, comprising: obtaining a test text output by the large model, wherein the large model performs a generative task to obtain the test text based on user input information; matching the test text with a preset trie to obtain a matching result, wherein multiple nodes of the preset trie each represent multiple preset characters related to preset abnormal words, the edge relationships between multiple nodes represent the positional relationships between the nodes and multiple preset characters in a preset string, and the preset string represents preset abnormal words; and determining abnormal words with semantic violation risks from the test text based on the matching result.

[0006] According to an embodiment of this application, the text to be tested is matched with a preset trie to obtain a matching result, including: matching multiple characters to be tested in the text to be tested with multiple nodes in the preset trie to obtain a path matching result. The path matching result represents the target preset string represented by the target path in the preset trie, which matches the target string to be tested in the text to be tested. The target path includes multiple preset characters in the target preset string and the positional relationship between the multiple preset characters.

[0007] According to an embodiment of this application, multiple test characters of the text to be tested are matched with multiple nodes of a preset trie to obtain a path matching result, including: matching the (n-1)th test character with the (n-1)th node corresponding to the (n-1)th test character to obtain a character matching result for the (n-1)th test character; if the character matching result of the (n-1)th test character indicates that the (n-1)th test character and the (n-1)th node are matched, determining the nth node that has an edge relationship with the (n-1)th node, wherein the hierarchical relationship of the nth node is lower than that of the (n-1)th node; and matching the nth node with the nth test character to obtain a character matching result for the nth test character.

[0008] According to an embodiment of this application, matching multiple test characters of the text to be tested with multiple nodes of a preset trie to obtain a path matching result further includes: when the character matching result of the (n-1)th test character indicates that the (n-1)th test character and the (n-1)th node do not match, determining a node combination based on the (n-1)th associated node that has an edge relationship with the (n-1)th node; determining an associated node combination that matches the node combination from the preset trie, wherein the field represented by the associated node combination is the same as the field represented by the node combination; and matching the (n-1)th node with the (n-1)th associated node corresponding to the associated node combination to obtain a target character matching result indicating that they match.

[0009] According to embodiments of this application, the above method further includes: segmenting the text to be tested to obtain multiple words to be tested; performing hash calculations on the multiple words to be tested using multiple hash functions to obtain the hash values ​​of each word to be tested; performing hash calculations on multiple preset abnormal words using multiple hash functions to obtain the preset hash values ​​of each preset abnormal word; matching the multiple hash values ​​to be tested with the multiple preset hash values ​​to obtain a hash value matching result; using the hash value matching result to represent the hash value to be tested that is consistent with the preset hash value as the target hash value, and updating the text to be tested based on the words to be tested indicated by the target hash value to obtain the updated text to be tested; wherein, matching the text to be tested with a preset trie to obtain a matching result includes: matching the updated text to be tested with the preset trie to obtain a matching result.

[0010] According to an embodiment of this application, the above method further includes: determining a target thread from multiple thread pools that matches the number of texts in the text to be tested, based on the number of idle threads in the thread pool, wherein the target thread is used to match the text to be tested with a preset trie to obtain a matching result.

[0011] According to an embodiment of this application, determining a target thread that matches the text quantity of the text to be tested from multiple thread pools based on the number of idle threads in the thread pool includes: determining multiple target threads that match the text quantity from multiple idle threads when the number of idle threads meets a preset threshold; or, when the number of idle threads does not meet the preset threshold, creating a specified number of new idle threads in the thread pool based on the text quantity of the text to be tested, and determining multiple target threads from the idle threads in the current thread pool.

[0012] According to an embodiment of this application, the method further includes: identifying abnormal input information among multiple input information of a large model and abnormal words in the test text corresponding to the multiple input information; determining the evaluation result of the large model based on the number of input information and the number of abnormal input information; and optimizing the large model based on the evaluation result of the large model to obtain an optimized large model.

[0013] A second aspect of this application provides an apparatus for detecting the output text of a large model, comprising: an acquisition module for acquiring the test text output by the large model, wherein the large model performs a generative task based on user input information to obtain the test text; a matching module for matching the test text with a preset trie to obtain a matching result, wherein multiple nodes of the preset trie each represent multiple preset characters related to preset abnormal words, the edge relationships between multiple nodes represent the positional relationships between the nodes and multiple preset characters in a preset string, and the preset string represents the preset abnormal words; and a determination module for determining abnormal words with semantic violation risks from the test text based on the matching result.

[0014] A third aspect of this application provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0015] A fourth aspect of this application also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0016] The fifth aspect of this application also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method.

[0017] According to embodiments of this application, the text to be tested is matched with multiple nodes in a preset trie that represent preset abnormal words. Matching results are obtained based on the edge relationships between the representations of multiple nodes and the positional relationships between multiple preset characters in a preset string. This allows for the identification of abnormal words with semantic violation risks from the text to be tested based on the matching results. Matching with a preset trie avoids the waste of human resources associated with manual testing. Furthermore, the association of multiple nodes in the preset trie with preset abnormal words enables rapid and accurate location of abnormal words with semantic violation risks in the text to be tested during the matching process, thereby improving both testing efficiency and accuracy. Attached Figure Description

[0018] The above-mentioned contents, other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0019] Figure 1 The illustration schematically depicts an application scenario of a method, apparatus, device, medium, and program product for detecting output text of a large model according to embodiments of this application.

[0020] Figure 2 A flowchart illustrating a method for detecting output text of a large model according to an embodiment of this application is shown schematically.

[0021] Figure 3 A schematic diagram of a preset dictionary tree according to an embodiment of this application is shown;

[0022] Figure 4 The illustration shows a schematic diagram of matching a preset dictionary tree according to an embodiment of this application;

[0023] Figure 5 A schematic diagram of a thread pool according to an embodiment of this application is shown;

[0024] Figure 6 This illustration schematically shows a diagram of detecting the output text of a large model according to an embodiment of this application;

[0025] Figure 7 This schematically illustrates a structural block diagram of an apparatus for detecting output text of a large model according to an embodiment of this application; and

[0026] Figure 8 A block diagram schematically illustrates an electronic device suitable for implementing a method for detecting output text of a large model according to an embodiment of this application. Detailed Implementation

[0027] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0028] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0029] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0030] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0031] In the technical solution of this application, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse.

[0032] In scenarios involving automated decision-making using personal information, the methods, devices, and systems provided in this application all offer users corresponding entry points for choosing to agree to or reject the automated decision-making results. If the user chooses to reject, the process proceeds to the expert decision-making stage. Here, "automated decision-making" refers to the activity of automatically analyzing and evaluating an individual's behavioral habits, interests, or economic, health, and credit status through computer programs, and then making a decision. Here, "expert decision-making" refers to the activity of making decisions by personnel who specialize in a particular field, possess specialized experience, knowledge, and skills, and have reached a certain level of professional expertise.

[0033] This application provides a method for detecting the output text of a large model. The method involves obtaining the test text output by the large model, wherein the large model performs a generative task based on user input to obtain the test text; matching the test text with a preset trie to obtain matching results, where multiple nodes of the preset trie each represent multiple preset characters related to preset abnormal words, and the edge relationships between multiple nodes represent the positional relationships between them and multiple preset characters in a preset string, where the preset string represents the preset abnormal words; and determining abnormal words with semantic violation risks from the test text based on the matching results.

[0034] Figure 1 The illustration schematically depicts an application scenario of a method, apparatus, device, medium, and program product for detecting output text of a large model according to embodiments of this application.

[0035] like Figure 1 As shown, application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0036] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0037] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0038] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0039] It should be noted that the method for detecting the output text of a large model provided in this application embodiment can generally be executed by server 105. Correspondingly, the apparatus for detecting the output text of a large model provided in this application embodiment can generally be located in server 105. The method for detecting the output text of a large model provided in this application embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the apparatus for detecting the output text of a large model provided in this application embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0040] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0041] The following will be based on Figure 1 The described scene, through Figures 2-6 A method for detecting the output text of a large model according to embodiments of this application will be described in detail.

[0042] Figure 2 The flowchart illustrates a method for detecting the output text of a large model according to an embodiment of this application.

[0043] like Figure 2 As shown, the detection of the output text of the large model in this embodiment includes operations S210 to S230.

[0044] In operation S210, the test text output by the large model is obtained. The large model performs a generative task based on the user's input information to obtain the test text.

[0045] Large Language Models (LLMs) can output results that match the user's input information, i.e., the test text output by the large model.

[0046] After obtaining the test text output by the large model, a thread can be created to process the test text, and subsequent operations S220 can be performed through this thread. When there are multiple different test texts, multiple different threads can execute operation S220 in parallel.

[0047] In operation S220, the text to be tested is matched with a preset trie to obtain the matching result. Each node of the preset trie represents a preset character related to a preset abnormal word. The edge relationship between multiple nodes represents the positional relationship between the preset characters in the preset string. The preset string represents the preset abnormal word.

[0048] A default trie can be a tree structure consisting of a root node and multiple nodes, where each node, except for the root node, indicates a character, which can be a letter or a literal.

[0049] Preset abnormal words can be words that are considered "unsafe responses to large models", such as words that do not conform to preset rules.

[0050] The multiple nodes of the preset trie indicate multiple preset characters of multiple preset abnormal words. Among them, the nodes with edge relationships have a hierarchical relationship from top to bottom according to the positional relationship between multiple preset characters in the preset string corresponding to the preset abnormal word. That is, the multiple first-level nodes with edge relationships with the root node are the first characters in the preset strings corresponding to the multiple preset abnormal words.

[0051] Matching the text to be tested with a preset trie can be done by matching each character of the text to multiple nodes of the preset trie, determining whether the words formed by the multiple characters of the text to be tested contain words that are consistent with preset abnormal words recorded in the preset trie, and obtaining the matching result.

[0052] In addition, after the matching results are generated, the corresponding threads can be deleted to reduce resource consumption.

[0053] By matching the text to be tested against a pre-defined trie, automated traversal of thousands / tens of thousands of test questions is achieved within minutes, speeding up testing by 10–100 times compared to manual testing. Furthermore, the concurrent stress testing conducted by multiple threads during the matching operation simulates real high-load scenarios, exposing security vulnerabilities in large models under peak request conditions (such as matching mechanism failure due to resource contention).

[0054] In operation S230, based on the matching results, abnormal words with semantic violation risks are identified from the text to be tested.

[0055] After obtaining the matching results, it is determined that there are abnormal words in the text to be tested that are consistent with the preset abnormal words recorded in the preset dictionary tree.

[0056] Once anomalous words are identified, large models can be optimized based on these anomalous words to reduce the probability of them generating anomalous words.

[0057] Furthermore, based on identified anomalous words, a security heatmap can be output (e.g., the heatmap shows a 25% failure rate for intercepting financial issues and a 12% failure rate for intercepting anomalous speech). This heatmap can pinpoint high-frequency failure types to guide security optimization strategies for large-scale model optimization.

[0058] According to embodiments of this application, the text to be tested is matched with multiple nodes in a preset trie that represent preset abnormal words. Matching results are obtained based on the edge relationships between the representations of multiple nodes and the positional relationships between multiple preset characters in a preset string. This allows for the identification of abnormal words with semantic violation risks from the text to be tested based on the matching results. Matching with a preset trie avoids the waste of human resources associated with manual testing. Furthermore, the association of multiple nodes in the preset trie with preset abnormal words enables rapid and accurate location of abnormal words with semantic violation risks in the text to be tested during the matching process, thereby improving both testing efficiency and accuracy.

[0059] According to an embodiment of this application, the text to be tested is matched with a preset trie to obtain a matching result, including: matching multiple characters to be tested in the text to be tested with multiple nodes in the preset trie to obtain a path matching result. The path matching result represents the target preset string represented by the target path in the preset trie, which matches the target string to be tested in the text to be tested. The target path includes multiple preset characters in the target preset string and the positional relationship between the multiple preset characters.

[0060] The character to be tested can be any word in the text to be tested. Matching multiple characters to be tested with multiple nodes of a preset trie can be done by matching multiple characters to be tested with preset characters indicated by multiple nodes character by character. During the matching process, based on the path between multiple nodes, multiple characters to be tested that can match a specified path are determined to obtain the path matching result.

[0061] The target path can be a path between multiple nodes indicated by the path matching result. The preset characters indicated by the multiple nodes of the target path constitute the target preset string. The order of the preset characters in the target preset string is related to the positional relationship between the multiple preset characters.

[0062] After obtaining multiple path matching results, the target preset string indicated by the multiple path matching results can be identified as the matching result.

[0063] According to the embodiments of this application, by matching multiple characters to be tested in the text to be tested with multiple nodes of a preset trie, it is possible to match character by character, thereby determining the target path between multiple nodes based on the matching between characters, which can improve the accuracy of matching and avoid the omission of abnormal words.

[0064] According to an embodiment of this application, multiple test characters of the text to be tested are matched with multiple nodes of a preset trie to obtain a path matching result, including: matching the (n-1)th test character with the (n-1)th node corresponding to the (n-1)th test character to obtain a character matching result for the (n-1)th test character; if the character matching result of the (n-1)th test character indicates that the (n-1)th test character and the (n-1)th node are matched, determining the nth node that has an edge relationship with the (n-1)th node, wherein the hierarchical relationship of the nth node is lower than that of the (n-1)th node; and matching the nth node with the nth test character to obtain a character matching result for the nth test character.

[0065] Match the (n-1)th character to be tested with the (n-1)th node corresponding to the (n-1)th character to be tested, and determine whether the (n-1)th character to be tested is consistent with the preset character indicated by the (n-1)th node. If they are consistent, the character matching result of the (n-1)th character to be tested represents the match between the (n-1)th character to be tested and the (n-1)th node.

[0066] If the (n-1)th character to be tested matches the (n-1)th node, the next level node of the (n-1)th node is determined based on the edge relationship of the (n-1)th node, and the nth node is determined.

[0067] After determining the nth node, determine the next character to be tested after the (n-1)th character to be tested, i.e. the nth character to be tested. Determine whether the nth character to be tested is consistent with the preset character indicated by the nth node, so as to obtain the character matching result of the nth character to be tested.

[0068] In some embodiments, for the first character to be tested, starting from the root node of the preset trie, a first node is determined among the nodes at the first level. The first character to be tested is compared with the preset character indicated by the first node to determine whether the first character to be tested and the preset character indicated by the first node are consistent. If there are multiple nodes at the first level, the first node can be determined from multiple nodes according to a preset traversal order, or it can be determined randomly from multiple nodes.

[0069] If the first character to be tested matches the preset character indicated by the first node, determine the second character to be tested after the first character to be tested, and determine the second node at the second level according to the edge relationship of the first node, so as to match the second character to be tested and the second node.

[0070] After all the test characters in the text to be tested have been matched, the path matching results of multiple test characters and multiple nodes of the preset trie can be determined based on the character matching results of all test characters.

[0071] According to the embodiments of this application, after the character to be tested and the node corresponding to the character to be tested are successfully matched, the node corresponding to the next character to be tested can be found through the edge relationship and hierarchical relationship between the nodes. In this way, the positional relationship between multiple preset characters can be accurately located during the matching process, thereby improving the accuracy of matching and avoiding the omission of abnormal words.

[0072] According to an embodiment of this application, matching multiple test characters of the text to be tested with multiple nodes of a preset trie to obtain a path matching result further includes: when the character matching result of the (n-1)th test character indicates that the (n-1)th test character and the (n-1)th node do not match, determining a node combination based on the (n-1)th associated node that has an edge relationship with the (n-1)th node; determining an associated node combination that matches the node combination from the preset trie, wherein the field represented by the associated node combination is the same as the field represented by the node combination; and matching the (n-1)th test character with the (n-1)th associated node corresponding to the associated node combination to obtain a target character matching result indicating that they match.

[0073] The character matching result of the (n-1)th character to be tested indicates that the (n-1)th character to be tested does not match the (n-1)th node, that is, the (n-1)th character to be tested is inconsistent with the preset character indicated by the (n-1)th node.

[0074] The (n-1)th associated node, which has an edge relationship with the (n-1)th node, can be multiple parent nodes that have an edge relationship with the (n-1)th node, and all of these parent nodes participate in the matching. For example, if the character matching result of the third test character indicates that the third test character does not match the third node, the parent node of the third node is identified. Then, the second node that matches the second test character is identified among the parent nodes of the third node. The parent node of the second node is further identified. Then, the first node that matches the first test character is identified among the parent nodes of the second node. Thus, the second node and the first node are determined as a node combination.

[0075] After determining the node combination, the associated node combination that matches the node combination is determined in the preset dictionary tree. Based on the (n-1)th associated node indicated by the associated node combination, the (n-1)th character to be tested is re-matched to obtain the character matching result of the new (n-1)th character to be tested.

[0076] In some embodiments, if the character matching results of the first character to be tested with multiple nodes in the first level all indicate a matching failure, then the second character to be tested is matched with multiple nodes in the first level.

[0077] In some embodiments, if the character matching result of the (n-1)th character to be tested represents that the (n-1)th character to be tested matches the (n-1)th node, and the (n-1)th node has no lower-level node, then the node combination related to the (n-1)th node is determined, so as to determine the associated node combination that matches the node combination from the preset dictionary based on the node combination related to the (n-1)th node, and then continue to match the nth character to be tested based on the node path indicated by the associated node combination.

[0078] In some embodiments, if the character matching result of the (n-1)th character to be tested indicates that the (n-1)th character to be tested matches the (n-1)th node, and the (n-1)th node has the end marker of the specified word, then the specified word is recorded.

[0079] In some embodiments, if a matching combination of nodes cannot be determined from the preset trie, the process jumps back to the root node to continue matching.

[0080] Figure 3 A schematic diagram of a preset dictionary tree according to an embodiment of this application is shown.

[0081] For example, suppose the text to be tested is "usher", such as Figure 3 As shown, assume that the default dictionary is composed of the default exception words “she”, “he”, “his” and “her”.

[0082] The first character to be tested in the text “usher” is “u”. Starting from the root node, the first level nodes are “s” and “h”, neither of which can match the first character to be tested “u”. Therefore, the second character to be tested is determined to be “s”.

[0083] If the second character to be tested, “s”, matches the node “s” in the first level, then the third character to be tested, “h”, is determined. Based on the edge relationship of the node “s” in the first level, if the third character to be tested, “h”, matches the node “h” in the second level, then the fourth character to be tested, “e”, is determined.

[0084] Based on the edge relationship of the second-level node "h", in the third level, it is determined that the fourth character to be tested, "e", matches the third-level node "e", thus determining the fifth character to be tested, "r". At the same time, the third-level node "e" indicates the end marker of "she", so "she" is recorded. The third-level node "e" indicates the end marker of "he", so "he" is recorded.

[0085] If the third-level node "e" has no lower-level node, then the node combination of the third-level node "e" is determined to be "she", "he" and "e". Furthermore, the associated node combination "he" composed of the first-level node "h" and the second-level node "e" can be determined.

[0086] Further determine the edge relationship of the second-level node "e", and determine the match between the fifth character to be tested "r" and the third-level node "r" in the third level to complete the matching of the text to be tested "usher". At the same time, the third-level node "r" indicates the end marker of "her", and "her" is recorded.

[0087] In the above matching process, based on the character matching results of all the characters to be tested, “he”, “she” and “her” are determined, and the paths indicated by “he”, “she” and “her” are determined as the path matching results of multiple characters to be tested in the text to be tested and multiple nodes of the preset dictionary tree.

[0088] According to the embodiments of this application, after the matching of the character to be tested and the node corresponding to the character to be tested fails, the associated node combination can be determined in the preset dictionary tree by determining the node combination, thereby avoiding re-starting the matching from the root node and improving the matching efficiency.

[0089] According to embodiments of this application, the above method further includes: segmenting the text to be tested to obtain multiple words to be tested; performing hash calculations on the multiple words to be tested using multiple hash functions to obtain the hash values ​​of each word to be tested; performing hash calculations on multiple preset abnormal words using multiple hash functions to obtain the preset hash values ​​of each preset abnormal word; matching the multiple hash values ​​to be tested with the multiple preset hash values ​​to obtain a hash value matching result; using the hash value matching result to represent the hash value to be tested that is consistent with the preset hash value as the target hash value, and updating the text to be tested based on the words to be tested indicated by the target hash value to obtain the updated text to be tested; wherein, matching the text to be tested with a preset trie to obtain a matching result includes: matching the updated text to be tested with the preset trie to obtain a matching result.

[0090] Multiple hash functions are used to perform hash calculations on multiple words to be tested, resulting in the individual hash values ​​of each word. The calculation results of the hash functions are unique, and different words to be tested indicate different hash values.

[0091] After calculating the test hash values ​​of multiple test words and the preset hash values ​​of multiple preset abnormal words, the test hash values ​​and preset hash values ​​are compared to determine whether they are consistent, so as to obtain the hash value matching result.

[0092] After obtaining the hash value matching result, the target hash value that matches the preset hash value is determined. Based on the target hash value, the target word is determined from multiple words to be tested and deleted from the text to be tested, thereby obtaining the updated text to be tested.

[0093] After determining the updated test text, simply match the updated test text with the preset trie to obtain the matching result.

[0094] Figure 4 The illustration shows a schematic diagram of matching a preset dictionary tree according to an embodiment of this application.

[0095] like Figure 4 As shown, in operation S410, the text to be tested is segmented into multiple words to be tested. Further, in operation S420, the hash values ​​of each of the multiple words to be tested are matched with the preset hash values ​​of each of the multiple preset abnormal words, and it is determined whether the hash values ​​to be tested and the preset hash values ​​match. If they match, the word to be tested indicated by the matched hash value is output as the matching result; if they do not match, operation S430 is performed to match one word to be tested with a preset trie to obtain the matching result.

[0096] According to the embodiments of this application, by comparing the test hash values ​​of multiple test words and the preset hash values ​​of multiple preset abnormal words, multiple test words can be pre-screened before matching with the preset trie. This can quickly exclude words that are absolutely not in the preset abnormal word set, thereby reducing the matching complexity when matching with the preset trie and improving matching efficiency.

[0097] According to an embodiment of this application, the above method further includes: determining a target thread from multiple threads that matches the number of texts in the text to be tested, based on the number of idle threads in the thread pool, wherein the target thread is used to match the text to be tested with a preset trie to obtain a matching result.

[0098] If there are idle threads in the thread pool, determine the number of idle threads and check if the number of idle threads supports the number of texts to be tested. If it does, assign a target thread that matches the number of texts to be tested, and then use the target thread to match the text to be tested with the preset trie.

[0099] If there are multiple target threads, the matching tasks are executed in parallel, thus enabling the processing of multiple texts simultaneously.

[0100] In some embodiments, the thread pool size can be dynamically adjusted for large models of different sizes.

[0101] Figure 5 A schematic diagram of a thread pool according to an embodiment of this application is shown.

[0102] like Figure 5 As shown, the thread pool contains multiple threads, each of which executes a different task. For example, thread 1 executes task A, thread 2 executes task B, and so on.

[0103] According to the embodiments of this application, since multiple target threads are set to perform matching operations simultaneously, there is no need for multiple texts to wait for each other, avoiding the inefficiency problem caused by a single thread, and also improving the utilization rate of idle threads in the thread pool.

[0104] According to an embodiment of this application, determining a target thread that matches the text quantity of the text to be tested from multiple thread pools based on the number of idle threads in the thread pool includes: determining multiple target threads that match the text quantity from multiple idle threads when the number of idle threads meets a preset threshold; or, when the number of idle threads does not meet the preset threshold, creating a specified number of new idle threads in the thread pool based on the text quantity of the text to be tested, and determining multiple target threads from the idle threads in the current thread pool.

[0105] If the number of idle threads meets the preset threshold, that is, if the task load indicated by the number of idle threads reaches the upper limit of the load of multiple threads executing tasks, then based on the current number of idle threads, multiple target threads that match the number of texts are determined from the multiple idle threads.

[0106] If the number of idle threads does not meet the preset threshold, new idle threads can be created within the preset threshold range to identify multiple target threads in the current thread pool based on the newly created idle threads.

[0107] In some embodiments, the target thread can be deleted after completing the matching operation, meaning there are no idle threads, thus reducing resource consumption. A new idle thread is created upon receiving new text to be tested.

[0108] In some embodiments, if the number of target threads reaches the upper limit and there are still texts without assigned target threads, a waiting queue is created to accommodate the excess thread requests.

[0109] According to embodiments of this application, by controlling the number of idle threads, resource consumption can be reduced and idle threads can be prevented from occupying too many resources.

[0110] According to an embodiment of this application, the method further includes: identifying abnormal input information among multiple input information of a large model and abnormal words in the test text corresponding to the multiple input information; determining the evaluation result of the large model based on the number of input information and the number of abnormal input information; and optimizing the large model based on the evaluation result of the large model to obtain an optimized large model.

[0111] For any set of input information and the corresponding test text pair containing anomalous words, if the result obtained by the large model after processing the input information contains anomalous words, then the input information is identified as anomalous input information.

[0112] After identifying abnormal input information, the number of input information and the number of abnormal input information are counted, and the ratio between the number of input information and the number of abnormal input information is determined. Based on this ratio, the validity of the input information of the large model is determined, and the evaluation results related to the security of the large model in processing input information can be analyzed based on the validity of the input information.

[0113] After determining the evaluation results of the large model, the large model is optimized based on these results so that it can filter effective information and output safe results, resulting in an optimized large model.

[0114] According to embodiments of this application, by analyzing multiple input information of a large model and abnormal words in the test text corresponding to the multiple input information, the large model can be reasonably optimized based on the analysis results, thereby improving the security of the output results of the large model.

[0115] Figure 6 The illustration shows a schematic diagram of detecting the output text of a large model according to an embodiment of this application.

[0116] like Figure 6As shown, in operation S610, the test text output by the large model is obtained. In operation S620, the target thread is determined. In operation S630, the test text is matched with a preset trie. In operation S640, based on the matching results, abnormal words with semantic violation risks are identified. In operation S650, a list of issues is output based on the abnormal words. The preset trie in operation S630 is determined based on operations S660 and S670. In operation S660, preset abnormal words are defined, and in operation S670, a preset trie is constructed based on the preset abnormal words.

[0117] Based on the above method for detecting the output text of large models, this application also provides an apparatus for detecting the output text of large models. The following will combine... Figure 7 The device is described in detail.

[0118] Figure 7 The diagram schematically illustrates a structural block diagram of an apparatus for detecting the output text of a large model according to an embodiment of this application.

[0119] like Figure 7 As shown, the apparatus 700 for detecting the output text of a large model in this embodiment includes an acquisition module 710, a matching module 720, and a determination module 730.

[0120] The acquisition module 710 is used to acquire the test text output by the large model, wherein the large model performs a generative task based on the user's input information to obtain the test text. In one embodiment, the acquisition module 710 can be used to perform the operation S210 described above, which will not be repeated here.

[0121] The matching module 720 is used to match the text to be tested with a preset trie to obtain a matching result. Multiple nodes in the preset trie each represent multiple preset characters related to a preset abnormal word. The edge relationships between the multiple nodes represent the positional relationships between them and the multiple preset characters in a preset string. The preset string represents the preset abnormal word. In one embodiment, the matching module 720 can be used to perform the operation S220 described above, which will not be repeated here.

[0122] The determination module 730 is used to identify anomalous words with semantic violation risks from the text to be tested based on the matching results. In one embodiment, the determination module 730 can be used to perform the operation S230 described above, which will not be repeated here.

[0123] According to embodiments of this application, the text to be tested is matched with multiple nodes in a preset trie that represent preset abnormal words. Matching results are obtained based on the edge relationships between the representations of multiple nodes and the positional relationships between multiple preset characters in a preset string. This allows for the identification of abnormal words with semantic violation risks from the text to be tested based on the matching results. Matching with a preset trie avoids the waste of human resources associated with manual testing. Furthermore, the association of multiple nodes in the preset trie with preset abnormal words enables rapid and accurate location of abnormal words with semantic violation risks in the text to be tested during the matching process, thereby improving both testing efficiency and accuracy.

[0124] According to an embodiment of this application, the matching module 720 includes a matching submodule.

[0125] The matching submodule is used to match multiple test characters in the test text with multiple nodes in the preset dictionary tree to obtain path matching results. The path matching results represent the target preset string represented by the target path in the preset dictionary tree, which matches the target test string in the test text. The target path includes multiple preset characters in the target preset string and the positional relationship between the multiple preset characters.

[0126] According to an embodiment of this application, the matching submodule includes a first matching unit, a first determining unit, and a second matching unit.

[0127] The first matching unit is used to match the (n-1)th character to be tested with the (n-1)th node corresponding to the (n-1)th character to be tested, and obtain the character matching result of the (n-1)th character to be tested.

[0128] The first determining unit is used to determine the nth node that has an edge relationship with the nth node when the character matching result of the n-1th test character represents a match between the n-1th test character and the n-1th node. The hierarchical relationship of the nth node is lower than that of the n-1th node.

[0129] The second matching unit is used to match the nth node with the nth character to be tested, and obtain the character matching result of the nth character to be tested.

[0130] According to embodiments of this application, the matching submodule further includes a second determining unit, a third determining unit, and a third matching unit.

[0131] The second determining unit is used to determine the node combination based on the (n-1)th node and the (n-1)th associated node that has an edge relationship with the (n-1)th node when the character matching result of the (n-1)th test character represents that the (n-1)th test character and the (n-1)th node do not match.

[0132] The third determining unit is used to determine the associated node combination that matches the node combination from the preset dictionary tree, wherein the fields represented by the associated node combination are the same as the fields represented by the node combination.

[0133] The third matching unit is used to match the (n-1)th node with the nth associated node corresponding to the associated node combination, and obtain the target character matching result that represents the matching.

[0134] According to an embodiment of this application, the apparatus 700 for detecting the output text of a large model includes a word segmentation module, a first calculation module, a second calculation module, a first matching module, a first update module, and a second matching module.

[0135] The word segmentation module is used to segment the text to be tested, resulting in multiple words to be tested.

[0136] The first calculation module is used to perform hash calculations on multiple words to be tested using multiple hash functions, so as to obtain the hash values ​​of each word to be tested.

[0137] The second calculation module is used to perform hash calculations on multiple preset abnormal words using multiple hash functions to obtain the preset hash values ​​of each preset abnormal word.

[0138] The first matching module is used to match multiple hash values ​​to be tested with multiple preset hash values ​​to obtain hash value matching results.

[0139] The first update module is used to take the hash value to be tested that matches the preset hash value as the target hash value, and update the text to be tested based on the words to be tested indicated by the target hash value to obtain the updated text to be tested.

[0140] The second matching module is used to match the updated test text with the preset trie to obtain the matching result.

[0141] According to an embodiment of this application, the apparatus 700 for detecting the output text of a large model includes a first determining module.

[0142] The first determining module is used to determine the target thread that matches the number of texts in the test text from multiple threads based on the number of idle threads in the thread pool. The target thread is used to match the test text with a preset trie to obtain the matching result.

[0143] According to an embodiment of this application, the first determining module includes a first determining submodule and a first creating submodule.

[0144] The first determination submodule is used to determine multiple target threads that are matched with the number of texts from multiple idle threads, provided that the number of idle threads meets a preset threshold.

[0145] The first creation submodule is used to create a specified number of new idle threads in the thread pool based on the number of texts to be tested when the number of idle threads does not meet the preset threshold, and to determine multiple target threads from the idle threads in the current thread pool.

[0146] According to an embodiment of this application, the apparatus 700 for detecting the output text of a large model includes a second determining module, a third determining module, and an optimization module.

[0147] The second determination module is used to identify abnormal input information from multiple input information of a large model and abnormal words in the test text corresponding to the multiple input information.

[0148] The third determination module is used to determine the evaluation result of the large model based on the number of input information and the number of abnormal input information.

[0149] The optimization module is used to optimize the large model based on the evaluation results of the large model, and obtain the optimized large model.

[0150] According to embodiments of this application, any plurality of modules among the acquisition module 710, matching module 720, and determination module 730 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least a portion of the functionality of one or more of these modules can be combined with at least a portion of the functionality of other modules and implemented in one module. According to embodiments of this application, at least one of the acquisition module 710, matching module 720, and determination module 730 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any appropriate combination of any of these three implementation methods. Alternatively, at least one of the acquisition module 710, matching module 720, and determination module 730 can be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0151] Figure 8 A block diagram schematically illustrates an electronic device suitable for implementing a method for detecting output text of a large model according to an embodiment of this application.

[0152] like Figure 8As shown, an electronic device 800 according to an embodiment of this application includes a processor 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage portion 808 into a random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this application.

[0153] RAM 803 stores various programs and data required for the operation of electronic device 800. Processor 801, ROM 802, and RAM 803 are interconnected via bus 804. Processor 801 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 802 and / or RAM 803. It should be noted that the programs may also be stored in one or more memories other than ROM 802 and RAM 803. Processor 801 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in said one or more memories.

[0154] According to embodiments of this application, the electronic device 800 may further include an input / output (I / O) interface 805, which is also connected to a bus 804. The electronic device 800 may also include one or more of the following components connected to the input / output (I / O) interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the input / output (I / O) interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 810 as needed so that computer programs read from it can be installed into the storage section 808 as needed.

[0155] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0156] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this application, the computer-readable storage medium may include ROM 802 and / or RAM 803 and / or one or more memories other than ROM 802 and RAM 803 described above.

[0157] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code enables the computer system to implement the method for detecting output text of a large model provided in embodiments of this application.

[0158] When the computer program is executed by the processor 801, it performs the functions defined in the system / apparatus of this application embodiment. According to the embodiments of this application, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0159] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 809, and / or installed from a removable medium 811. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0160] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 809, and / or installed from the removable medium 811. When the computer program is executed by the processor 801, it performs the functions defined in the system of this application embodiment. According to the embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0161] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0162] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0163] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

Claims

1. A method for detecting the output text of a large model, characterized in that, The method includes: Obtain the test text output by a large model, wherein the large model performs a generative task based on user input information to obtain the test text; The text to be tested is matched with a preset trie to obtain a matching result. Each node of the preset trie represents a number of preset characters related to a preset abnormal word. The edge relationship between the nodes represents the positional relationship between the nodes and the preset characters in the preset string. The preset string represents the preset abnormal word. Based on the matching results, abnormal words with semantic violation risks are identified from the text to be tested.

2. The method according to claim 1, characterized in that, The step of matching the text to be tested with a preset trie to obtain the matching result includes: Multiple test characters of the text to be tested are matched with multiple nodes of the preset trie to obtain a path matching result. The path matching result represents the target preset string represented by the target path in the preset trie, which matches the target test string in the text to be tested. The target path includes multiple preset characters in the target preset string and the positional relationship between the multiple preset characters.

3. The method according to claim 2, characterized in that, The process of matching multiple test characters of the test text with multiple nodes of the preset trie to obtain path matching results includes: Match the (n-1)th character to be tested with the (n-1)th node corresponding to the (n-1)th character to be tested, and obtain the character matching result of the (n-1)th character to be tested, where n is an integer greater than 1; If the character matching result of the (n-1)th character to be tested indicates that the (n-1)th character to be tested matches the (n-1)th node, then determine the nth node that has an edge relationship with the (n-1)th node, and the hierarchical relationship of the nth node is lower than that of the (n-1)th node; The nth node and the nth character to be tested are matched to obtain the character matching result of the nth character to be tested.

4. The method according to claim 3, characterized in that, The step of matching multiple test characters of the test text with multiple nodes of the preset trie to obtain a path matching result further includes: In the case where the character matching result of the (n-1)th character to be tested does not match the (n-1)th character to be tested and the (n-1)th node, the node combination is determined based on the (n-1)th associated node that has an edge relationship with the (n-1)th node. Determine the associated node combination that matches the node combination from the preset trie, wherein the field represented by the associated node combination is the same as the field represented by the node combination; The (n-1)th character to be tested is matched with the (n-1)th associated node corresponding to the combination of the associated nodes to obtain the target character matching result that represents the match.

5. The method according to claim 1, characterized in that, The method further includes: The text to be tested is segmented into words to obtain multiple words to be tested; Multiple hash functions are used to perform hash calculations on multiple words to be tested, and the hash values ​​of each word to be tested are obtained. Multiple hash functions are used to perform hash calculations on the multiple preset abnormal words to obtain the preset hash values ​​of each of the multiple preset abnormal words; The hash values ​​to be tested and the preset hash values ​​are matched to obtain hash value matching results; The hash value matching result represents the test hash value that is consistent with the preset hash value as the target hash value, and the test text is updated based on the test word indicated by the target hash value to obtain the updated test text; The step of matching the text to be tested with a preset trie to obtain a matching result includes: The updated test text is matched with the preset trie to obtain the matching result.

6. The method according to claim 1, characterized in that, The method further includes: Based on the number of idle threads in the thread pool, a target thread is determined from the multiple threads that matches the number of texts in the text to be tested. The target thread is used to match the text to be tested with a preset trie to obtain a matching result.

7. The method according to claim 6, characterized in that, The step of determining a target thread from multiple thread pools that matches the number of texts in the text to be tested, based on the number of idle threads in the thread pool, includes: If the number of idle threads meets a preset threshold, determine multiple target threads that definitively match the number of text threads from among the multiple idle threads; or If the number of idle threads does not meet the preset threshold, a specified number of new idle threads are created in the thread pool based on the number of texts in the text to be tested, and multiple target threads are determined from the idle threads in the current thread pool.

8. The method according to claim 1, characterized in that, The method further includes: For the multiple input information of the large model and the abnormal words of the test text corresponding to the multiple input information, the abnormal input information is determined from the multiple input information; The evaluation result of the large model is determined based on the number of input information and the number of abnormal input information. Based on the evaluation results of the large model, the large model is optimized to obtain the optimized large model.

9. An apparatus for detecting the output text of a large model, characterized in that, The device includes: The acquisition module is used to acquire the test text output by the large model, wherein the large model performs a generative task based on the user's input information to obtain the test text; A matching module is used to match the text to be tested with a preset trie to obtain a matching result. Multiple nodes in the preset trie each represent multiple preset characters related to a preset abnormal word. The edge relationships between the multiple nodes represent the positional relationships between the nodes and the multiple preset characters in a preset string. The preset string represents the preset abnormal word. The determination module is used to identify abnormal words with semantic violation risks from the text to be tested based on the matching results.

10. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 8.

12. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 8.

Images