A data compliance detection method, apparatus and computing device cluster

Abstract

A data compliance detection method comprises: receiving a query question from a user; determining whether the query question is compliant based on the similarity between the query question and the content in a sample library, wherein the sample library contains user feedback missed samples and false positive samples. The method determines whether the query question is compliant by the similarity between the query question and the content in the sample library, which can realize the customization configuration of different user application scenarios, quickly adapt to different user application scenarios, reduce the probability of missed detection and false positive, and improve the accuracy of security detection. Through the sample library, the missed detection / false positive samples can be quickly updated, so as to quickly iterate according to the user demand, greatly reducing the high cost of retraining the model.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI) technology, and in particular to a data compliance detection method, device, and computing equipment cluster. Background Technology

[0002] With the widespread application of large language models (LLMs) in various scenarios such as banking and healthcare, their security issues have become increasingly prominent. To ensure the secure use of LLMs, current security solutions mainly include intrinsic model security mechanisms and external security models. Intrinsic model security mechanisms enhance the model's ability to identify unsafe content by embedding security features during model training, such as secure alignment, defensive neurons, and optimized training strategies. However, this approach is less effective when faced with complex unsafe queries. External security models, on the other hand, use techniques such as content filtering, inference guidance, and secure multi-party computation to perform real-time detection and filtering of input and output. However, this approach lacks adaptability to various scenarios and is prone to false positives, especially in healthcare settings where some normal and necessary content is misjudged as unsafe, severely impacting user experience. Furthermore, when serious unsafe content is missed or safe content is falsely flagged, the problem is usually fixed by retraining the external security model, which is time-consuming and fails to meet the rapid remediation requirements of application scenarios. Summary of the Invention

[0003] This application provides a data compliance detection method, apparatus, computing device cluster, computer-readable storage medium, and computer program product containing instructions, which can quickly adapt to different user application scenarios, reduce the probability of missed detections and false positives, improve the accuracy of security detection, and can be rapidly iterated according to user needs, greatly reducing the high cost of retraining models.

[0004] Firstly, this application provides a data compliance detection method, comprising: receiving a query question from a user; determining whether the query question is compliant based on the similarity between the query question and content in a sample library, wherein the sample library contains missed detection examples and false positive examples reported by the user.

[0005] In this way, by determining the compliance of a query by comparing its similarity to user feedback in the sample database, customized configurations can be achieved for different user application scenarios. This allows for rapid adaptation to various user scenarios, reducing the probability of missed detections and false positives, and improving the accuracy of security detection. The sample database also allows for rapid updates to missed / false positive examples, enabling quick iteration based on user needs and significantly reducing the high costs associated with retraining the model.

[0006] In one possible implementation, the compliance of a query is determined based on its similarity to content in the sample database. This includes determining that the query is non-compliant if it is similar to a missed detection example, and compliant if it is similar to a false positive example. This is how the compliance of a query can be determined.

[0007] In one possible implementation, the compliance of a query is determined based on its similarity to content in a sample database. This involves using content from the sample database as system prompts and the query as user prompts, inputting these prompts into a neural network model to determine compliance. This allows the neural network model to determine whether a query is compliant.

[0008] In one possible implementation, the method further includes: caching and compressing the processing results of each network layer in the neural network model on the content in the sample library; or loading the pre-stored processing results of each network layer in the neural network model on the content in the sample library. In this way, by compressing, pre-storing, and reusing the system prompt words corresponding to the black and white sample libraries, the storage and pre-inference time can be saved, thereby efficiently assisting the neural network model in identifying content similar to the black and white sample libraries, thus improving compliance detection efficiency.

[0009] In one possible implementation, the sample library comprises a black sample library and a white sample library. The black sample library contains missed detection examples, and the white sample library contains false positive examples. Before determining compliance based on the similarity between the query and the content in the sample libraries, the process includes: clustering the content in the black sample library to obtain N clusters, and clustering the content in the white sample library to obtain M clusters, where N and M are both greater than or equal to 1; performing semantic compression on the content in each of the N clusters, and performing semantic compression on the content in each of the M clusters, where semantic compression is used to reduce duplicate or similar content. Thus, by performing semantic compression on the content in both the black and white sample libraries, duplicate or similar content in both libraries can be reduced, thereby improving the efficiency of subsequent similarity matching and ultimately improving compliance detection efficiency.

[0010] In one possible implementation, the method further includes: receiving an update operation from a user on the sample library; and updating the contents of the sample library in response to the update operation.

[0011] Secondly, this application provides a data compliance detection device, including a receiving module and a processing module. The receiving module is used to receive queries from users. The processing module is used to determine whether the query is compliant based on the similarity between the query and content in a sample library, wherein the sample library contains missed detection examples and false positive examples reported by users.

[0012] In one possible implementation, when the processing module determines whether a query is compliant based on the similarity between the query question and the content in the sample library, it is specifically used to: determine that the query question is non-compliant if it is similar to a missed detection example; and determine that the query question is compliant if it is similar to a false positive example.

[0013] In one possible implementation, when the processing module determines whether a query is compliant based on the similarity between the query question and the content in the sample library, it specifically uses the content in the sample library as system prompt words and the query question as user prompt words to input into a neural network model to determine whether the query question is compliant.

[0014] In one possible implementation, the processing module is further configured to: cache and compress the processing results of each network layer in the neural network model on the content in the sample library; or, load the pre-stored processing results of each network layer in the neural network model on the content in the sample library.

[0015] In one possible implementation, the sample library includes a black sample library and a white sample library. The black sample library contains missed detection examples, and the white sample library contains false positive examples. Before determining whether a query is compliant based on the similarity between the query and the content in the sample libraries, the processing module further performs the following steps: clustering the content in the black sample library to obtain N clusters, and clustering the content in the white sample library to obtain M clusters, where N and M are both greater than or equal to 1; performing semantic compression on the content in each of the N clusters, and performing semantic compression on the content in each of the M clusters, where semantic compression is used to reduce duplicate or similar content.

[0016] In one possible implementation, the receiving module is further configured to receive user update operations on the sample library; the processing module is further configured to update the content in the sample library in response to the update operation.

[0017] Thirdly, embodiments of this application provide a data compliance detection device, including at least one processor and an interface; the at least one processor obtains program instructions through the interface; the at least one processor is used to execute program line instructions to implement the method described in the first aspect or any possible implementation of the first aspect. Exemplarily, the data compliance detection device may be, but is not limited to, a chip.

[0018] Fourthly, this application provides a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the method described in the first aspect or any possible implementation of the first aspect.

[0019] Fifthly, this application provides a computer-readable storage medium including computer program instructions that, when executed by a cluster of computing devices, perform the method described in the first aspect or any possible implementation thereof. Exemplarily, the computing device cluster may include one or more computing devices.

[0020] Sixthly, this application provides a computer program product containing instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method described in the first aspect or any possible implementation thereof. Exemplarily, the cluster of computing devices may include one or more computing devices.

[0021] It is understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0022] Figure 1 This is a schematic diagram illustrating a technical concept for data compliance detection provided in an embodiment of this application;

[0023] Figure 2 This is a flowchart illustrating a data compliance detection method provided in an embodiment of this application;

[0024] Figure 3 This is a schematic diagram illustrating another technical concept for data compliance detection provided in an embodiment of this application;

[0025] Figure 4 This is a schematic diagram illustrating the process of compressing the content in a white sample library according to an embodiment of this application;

[0026] Figure 5 This is a flowchart illustrating another data compliance detection method provided in an embodiment of this application;

[0027] Figure 6 This is a schematic diagram illustrating another technical concept for data compliance detection provided in an embodiment of this application;

[0028] Figure 7 This is a schematic diagram illustrating different compression methods applied to the outputs of different network layers in a neural network model, as provided in an embodiment of this application.

[0029] Figure 8 This is a flowchart illustrating another data compliance detection method provided in the embodiments of this application;

[0030] Figure 9 This is a schematic diagram of the structure of a data compliance detection device provided in an embodiment of this application;

[0031] Figure 10This is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0032] Figure 11 This is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application;

[0033] Figure 12 This is a schematic diagram of another computing device cluster structure provided in an embodiment of this application;

[0034] Figure 13 This is a schematic diagram of another data compliance detection device provided in the embodiments of this application. Detailed Implementation

[0035] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0036] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first response message" and "second response message," etc., are used to distinguish different response messages, not to describe a specific order of response messages.

[0037] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0038] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0039] First, the relevant technical terms used in this application will be introduced.

[0040] (1) Neural Network Model

[0041] A neural network model is a computational model inspired by the structure of the human brain. It processes information by mimicking the connections and interactions of neurons in the brain. In a neural network model, the network layer is the basic unit that constitutes the model; it is a hierarchical structure for processing data. Each network layer can consist of multiple neurons, which are responsible for receiving input, performing calculations, and passing the results to the next layer. Furthermore, each network layer can be responsible for performing a specific computational task. Network layers in a neural network model can include one or more of the following: input layer, hidden layer, output layer, fully connected layer, convolutional layer, pooling layer, self-attention layer, normalization layer, and multilayer perceptron (MLP). For example, a neural network model can be, but is not limited to, a deep neural network (DNN) model, a convolutional neural network (CNN) model, or a transformer model.

[0042] (2) Large Language Model

[0043] Large language models are artificial intelligence models based on deep learning techniques. They are trained on large-scale text datasets to learn the syntax, semantics, and pragmatics of a language. These models have numerous parameters and can generate high-quality language output from input text, such as answering questions, translating text, and creating stories, among other natural language processing tasks.

[0044] (3) Prompt

[0045] A Prompt is a text instruction used to guide a neural network model in generating specific content. Its core function is to provide the model with context and a task objective. To provide context, a Prompt helps the model understand the semantic environment of the input data by including relevant information or background. To specify the task objective, a Prompt tells the model the task to be performed through explicit instructions, such as "check for errors in the data and correct them" or "analyze trends in the data." This instruction guides the model to think and generate content in a specific direction.

[0046] (4) System Prompt and User Prompt

[0047] The System Prompt is a "hidden rule" used to define the personality, style, or response method of the neural network model, providing basic guiding principles. The User Prompt, on the other hand, is the specific question or task posed by the user, directly determining what the model needs to solve. During inference, the neural network model considers both the System Prompt and the User Prompt simultaneously, as follows: First, the neural network model loads the System Prompt, using it as the basic guiding principle for generating output; then, the neural network model reads the User Prompt, i.e., it reads the specific question or task input by the user; finally, the neural network model generates a satisfactory output based on the guiding principles of the System Prompt and the specific requirements of the User Prompt.

[0048] (5) Missed detection cases and false positive cases

[0049] A missed detection instance refers to a query that fails to detect an actual security threat or violation. In other words, when a query is indeed insecure, but fails to identify it as a threat and is incorrectly classified as "safe" or "normal," it is a missed detection instance. For example, in a medical context, a query containing violent or inappropriate content might not be identified and blocked, but instead be incorrectly classified as normal content.

[0050] A false positive is a query that mistakenly identifies normal, safe content or behavior as unsafe or illegal. In other words, when a query is inherently harmless but is identified as a threat, it is incorrectly classified as "unsafe" or "abnormal." For example, in a medical context, a query containing normal medical terminology such as "surgical details" might be mistakenly blocked or flagged as harmful content.

[0051] The technical solution provided in this application will be described below.

[0052] For example, Figure 1 This illustration shows a schematic diagram of a technical concept for data compliance detection provided in an embodiment of this application. Figure 1As shown, the framework for data compliance testing can include a blacklist / whitelist sample library construction component and a compliance testing component. The blacklist / whitelist sample library construction component primarily collects user-reported missed / false positive examples and forms the blacklist / whitelist. The blacklist sample library may mainly consist of missed examples, while the whitelist sample library may mainly consist of false positive examples. For example, missed / false positive examples can be provided by users, but are not limited to, to adapt to the user's required compliance testing application scenarios. For instance, users in the legal field can provide missed / false positive examples for legal scenarios, users in the medical field can provide missed / false positive examples for medical scenarios, users in the financial field can provide missed / false positive examples for financial scenarios, and so on.

[0053] The compliance detection section primarily performs compliance checks on user-input queries based on a blacklist / whitelist sample library. Both the blacklist / whitelist sample library and the query can be input into the compliance detection model. For example, the blacklist / whitelist sample library can be input as the System Prompt, and the query as the User Prompt. The compliance detection model uses its semantic understanding capabilities combined with the content of the blacklist / whitelist sample library to determine the compliance of the query. Specifically, if the query is similar to content in the blacklist sample library, it is considered non-compliant; if it is similar to content in the whitelist sample library, it is considered compliant. For example, the compliance detection model can be a large language model or other neural network models, depending on the specific circumstances, and is not limited here.

[0054] The following description Figure 1 The specific implementation process of the technical concept shown.

[0055] For example, Figure 2 This illustration shows a flowchart of a data compliance detection method provided in an embodiment of this application. It is understood that this method can be executed by any device, equipment, platform, or device cluster with computing and processing capabilities. For example, the method can be executed by a data compliance detection device, which can be implemented by software and / or hardware and can be, but is not limited to, configured in an electronic device or server. For ease of description, the following description uses execution by a computing device as an example. Figure 2 As shown, this data compliance detection method may include the following steps:

[0056] S201, Receive a query from the user.

[0057] In this embodiment, a query question refers to an inquiry or request made by a user to a computing device in natural language, aiming to obtain specific information, answers, or solutions to satisfy their cognitive or operational needs regarding a particular topic or situation. After the user enters the query question on the computing device and issues an answer command, the computing device can receive the query question.

[0058] S202. Based on the similarity between the query question and the content in the sample library, determine whether the query question is compliant. The sample library contains user-reported missed detection examples and false positive examples.

[0059] In this embodiment, the computing device can calculate the similarity between the query question and the content in the sample library using similarity algorithms such as cosine similarity and term frequency-inverse document frequency, and determine whether the query question is compliant based on the calculated similarity. The sample library contains user-reported missed detection examples and false positive examples. For example, when the similarity between the query question and a missed detection example is the highest, the query question can be determined to be non-compliant; when the similarity between the query question and a false positive example is the highest, the query question can be determined to be compliant. For example, the query question can be used as a User Prompt, and the content in the sample library can be used as a System Prompt, input into a neural network model to output whether the query question is compliant. In some embodiments, the sample library can be, but is not limited to, divided into a black sample library and a white sample library. The black sample library contains user-reported missed detection examples, and the white sample library contains user-reported false positive examples. This facilitates data management and storage and is suitable for parallel computing. In this case, the computing device can calculate the similarity between the query question and each piece of content in the black sample library, and also calculate the similarity between the query question and each piece of content in the white sample library.

[0060] In this way, by determining the compliance of a query by comparing its similarity to user feedback in the sample database, customized configurations can be achieved for different user application scenarios. This allows for rapid adaptation to various user scenarios, reducing the probability of missed detections and false positives, and improving the accuracy of security detection. The sample database also allows for rapid updates to missed / false positive examples, enabling quick iteration based on user needs and significantly reducing the high costs associated with retraining the model.

[0061] For example, Figure 3 This illustration shows a schematic diagram of another technical concept for data compliance detection provided in an embodiment of this application. For example... Figure 3 As shown, the framework for data compliance testing can include: a blacklist / whitelist database construction component, a blacklist / whitelist database compression component, and a compliance testing component. For the blacklist / whitelist database construction component and the compliance testing component, please refer to the aforementioned sections. Figure 1 The description in the text will not be repeated here.

[0062] The black and white sample database compression section primarily performs semantic compression on the content in the black and white sample databases respectively. This reduces duplicate or similar content in both databases, thereby improving the efficiency of subsequent similarity matching and ultimately enhancing compliance detection efficiency. For example, before inputting the content from the black and white sample databases into the compliance detection model, algorithms such as K-Means clustering can be used to divide the content into multiple clusters. Then, semantic compression is performed on each cluster using a large language model to reduce duplicate or similar content within each cluster. For instance, taking the compression of the white sample database as an example... Figure 4 As shown, when the white sample library contains false positive examples from different scenarios, the false positive examples can first be classified according to the scenario. Then, within a single scenario, the false positive examples are divided into multiple clusters based on their semantic similarity. Finally, each cluster is compressed using a large language model. For example, when compressing the content of a certain cluster, at least one example can be selected as the desired example. Additionally, during compression, keywords can be extracted from the examples to replace them, thereby further compressing the examples. For instance, when the examples are "how to compensate for patent infringement" and "what is the legal basis for patent infringement compensation," the compressed content could be "patent infringement, compensation, legal basis."

[0063] The following description Figure 3 The specific implementation process of the technical concept shown.

[0064] For example, Figure 5 This illustration shows a flowchart of another data compliance detection method provided in an embodiment of this application. It is understood that this method can be executed by any device, equipment, platform, or device cluster with computing and processing capabilities. For example, this method can be executed by a data compliance detection device, which can be implemented by software and / or hardware and can be, but is not limited to, configured in an electronic device or server. For ease of description, the following description uses execution by a computing device as an example. Figure 5 S503 and S504 can be referenced. Figure 2 The descriptions of S201 and S202 are not repeated here. Figure 5 As shown, this data compliance detection method may include the following steps:

[0065] S501. Cluster the contents of the black sample library to obtain N clusters, and cluster the contents of the white sample library to obtain M clusters, where N and M are both greater than or equal to 1. The black sample library contains missed detection samples reported by users, and the white sample library contains false positive samples reported by users.

[0066] In this embodiment, after dividing the black sample library and the white sample library, the computing device can use a clustering algorithm to cluster the contents of the black sample library to obtain N clusters, and cluster the contents of the white sample library to obtain M clusters, where N and M are both greater than or equal to 1. The clustering process can be found in the aforementioned description of the "Black and White Sample Library Compression Section," and will not be repeated here.

[0067] S502. Perform semantic compression on the content of each of the N clusters, and perform semantic compression on the content of each of the M clusters, wherein semantic compression is used to reduce duplicate or similar content.

[0068] In this embodiment, the computing device can perform semantic compression on the content of each of the N and M clusters to reduce duplicate or similar content in each cluster. The semantic compression process can be found in the aforementioned description of the "Black and White Sample Library Compression Section," and will not be repeated here.

[0069] S503, Receive queries from users.

[0070] S504. Based on the similarity between the query question and the content in the sample library, determine whether the query question is compliant. The sample library includes a black sample library and a white sample library.

[0071] In this way, by semantically compressing the content in the black and white sample databases, duplicate or similar content can be reduced, thereby improving the efficiency of subsequent similarity matching and ultimately enhancing compliance detection efficiency. Furthermore, Figure 5 The described compliance testing methods can also bring... Figure 2 For details on the effects of the described compliance testing methods, please refer to [link / reference]. Figure 2 The description in [the original text] will not be repeated here. Furthermore, it should be understood that... Figure 5 The sample library used by S504 is a compressed black and white sample library.

[0072] For example, Figure 6 This illustration shows a schematic diagram of another technical concept for data compliance detection provided in an embodiment of this application. For example... Figure 6 As shown, the framework for data compliance testing can include: a blacklist / whitelist database construction component, a blacklist / whitelist database compression component, a compliance testing component, and a compression and caching (or data reuse) component. For the blacklist / whitelist database construction component and the compliance testing component, please refer to the aforementioned sections. Figure 1 The description in [the previous text] will not be repeated here. For the section on compressing the black and white sample library, please refer to the aforementioned [reference]. Figure 3 The description in the text will not be repeated here.

[0073] The compression and caching (or data reuse) section is primarily used to cache the output results of each network layer of the compliance detection model for each sample in the blacklist / whitelist database during the compliance processing. Since the blacklist / whitelist database may contain a large amount of content, the inference latency of the compliance detection model can be affected. However, by caching the output results of each network layer of the compliance detection model for each sample in the blacklist / whitelist database while keeping the samples in the database unchanged, the cached results can be reused in subsequent compliance checks for other queries, thus significantly improving compliance detection efficiency. Additionally, when caching the output results of each network layer, the output results of each network layer can also be compressed to reduce storage space requirements. The compression method used for different network layers can be selected based on the actual situation. When the output results of two adjacent network layers for a sample do not change significantly, the output result of the latter network layer can be compressed on a large scale. This allows for maximum compression while preserving key information in the data to the greatest extent possible. For example, such as... Figure 7 As shown, when the network layers in the compliance detection model use Transformer layers, different compression methods can be used to compress the key-value states (KV states) of each Transformer layer. For example, caching compression techniques such as fast generation (FastGen), grouped query attention, and low-rank adaptive key-value cache (LoRA KV Cache) can be used for compression.

[0074] The following description Figure 6 The specific implementation process of the technical concept shown.

[0075] For example, Figure 8 This illustration shows a flowchart of another data compliance detection method provided in an embodiment of this application. It is understood that this method can be executed by any device, equipment, platform, or device cluster with computing and processing capabilities. For example, this method can be executed by a data compliance detection device, which can be implemented by software and / or hardware and can be, but is not limited to, configured in an electronic device or server. For ease of description, the following description uses execution by a computing device as an example. Figure 8 The S801 in the text can be referenced. Figure 2 The description of S201 will not be repeated here. Figure 8 As shown, this data compliance detection method may include the following steps:

[0076] S801, Receive queries from users.

[0077] S802. Using the content in the sample library as system prompt words and the query question as user prompt words, input the neural network model to determine whether the query question is compliant. The sample library contains user-reported missed detection examples and false positive examples. During the data processing process of the neural network model, the processing results of each network layer in the neural network model on the content in the sample library are cached and compressed, or the pre-stored processing results of each network layer in the neural network model on the content in the sample library are loaded.

[0078] In this embodiment, the computing device can use content from a sample library as system prompts and the query question as user prompts, and input both simultaneously or at different times into a neural network model. The neural network then determines whether the query question is compliant based on the similarity between the two. While the neural network model processes the content from the sample library, the computing device can cache and compress the processing results of each network layer in the neural network model for reuse in subsequent compliance checks. Alternatively, if the processing results of each network layer in the neural network model are pre-cached, the computing device can directly load these pre-cached results from storage. This allows for direct use of the processing results during processing, improving the inference latency of the neural network model and optimizing its memory requirements.

[0079] In this way, by compressing, pre-storing, and reusing the system prompts corresponding to the black and white sample databases, the storage and pre-inference time can be saved. This allows for efficient assistance to the neural network model in identifying content similar to that in the black and white sample databases, thereby improving compliance detection efficiency. Furthermore, Figure 8 The described compliance testing methods can also bring... Figure 2 For details on the effects of the described compliance testing methods, please refer to [link / reference]. Figure 2 The description in [the original text] will not be repeated here. Furthermore, it should be understood that... Figure 8 The sample library used in this process can also be a compressed black and white sample library. Figure 8 The described compliance testing methods can also bring... Figure 5 For details on the effects of the described compliance testing methods, please refer to [link / reference]. Figure 2 The description in the text will not be repeated here.

[0080] It is understood that the "black and white sample library construction part, black and white sample library compression part, compliance detection part, and compression and caching (or data reuse) part" described in the above embodiments can be selected one or more according to actual conditions, and the solution formed after selecting one or more is still within the protection scope of this application. In addition, the sequence number of each step in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. Furthermore, the various embodiments and features described in the above embodiments can be combined according to actual conditions, and the combined solution is still within the protection scope of this application.

[0081] Next, based on the methods in the above embodiments, a data compliance detection device provided in this application will be introduced.

[0082] For example, Figure 9 A schematic diagram of the structure of a data compliance detection device provided in an embodiment of this application is shown. Figure 9 As shown, the data compliance detection device 900 includes a receiving module 901 and a processing module 902. The receiving module 901 receives queries from users. The processing module 902 determines whether the query is compliant based on the similarity between the query and content in a sample database, wherein the sample database contains missed detection examples and false positive examples reported by users.

[0083] In some embodiments, when determining whether a query question is compliant based on the similarity between the query question and the content in the sample library, the processing module 902 is specifically used to: determine that the query question is non-compliant if it is similar to a missed detection sample; and determine that the query question is compliant if it is similar to a false positive sample.

[0084] In some embodiments, when the processing module 902 determines whether a query question is compliant based on the similarity between the query question and the content in the sample library, it is specifically used to: use the content in the sample library as system prompt words and the query question as user prompt words, input them into a neural network model to determine whether the query question is compliant.

[0085] In some embodiments, the processing module 902 is further configured to: cache and compress the processing results of each network layer in the neural network model on the content in the sample library; or, load the pre-stored processing results of each network layer in the neural network model on the content in the sample library.

[0086] In some embodiments, the sample library includes a black sample library and a white sample library. The black sample library contains missed detection examples, and the white sample library contains false positive examples. In this case, before determining whether the query is compliant based on the similarity between the query and the content in the sample library, the processing module 902 further performs the following: clustering the content in the black sample library to obtain N clusters, and clustering the content in the white sample library to obtain M clusters, where N and M are both greater than or equal to 1; performing semantic compression on the content in each of the N clusters, and performing semantic compression on the content in each of the M clusters, wherein semantic compression is used to reduce duplicate or similar content.

[0087] In some embodiments, the receiving module 901 is further configured to receive an update operation from a user on the sample library; the processing module 902 is further configured to update the content in the sample library in response to the update operation.

[0088] In some embodiments, Figure 9 Both the receiving module 901 and the processing module 902 shown can be implemented in software or in hardware. For example, the implementation of the receiving module 901 will be described below. Similarly, the implementation of the processing module 902 can be referenced from the implementation of the receiving module 901.

[0089] As an example of a software functional unit, the receiving module 901 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the aforementioned computing instance may be one or more. For example, the receiving module 901 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0090] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0091] As an example of a hardware functional unit, the receiving module 901 may include at least one computing device, such as a server. Alternatively, the receiving module 901 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The aforementioned PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0092] The multiple computing devices included in the receiving module 901 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the receiving module 901 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the receiving module 901 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0093] It should be noted that, in other embodiments, the receiving module 901 can be used to execute any step in the data compliance detection method described in the above embodiments, and the processing module 902 can also be used to execute any step in the data compliance detection method described in the above embodiments. Furthermore, the steps implemented by the receiving module 901 and the processing module 902 can also be specified as needed, and different steps in the data compliance detection method described in the above embodiments can be implemented by the receiving module 901 and the processing module 902 respectively. Figure 9 The data compliance testing device 900 shown has all the functions of the device.

[0094] This application also provides a computing device 1000. For example... Figure 10 As shown, the computing device 1000 includes a bus 1002, a processor 1004, a memory 1006, and a communication interface 1008. The processor 1004, the memory 1006, and the communication interface 1008 communicate with each other via the bus 1002. The computing device 1000 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 1000.

[0095] Bus 1002 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 10 The bus 1004 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 1004 may include a path for transmitting information between various components of the computing device 1000 (e.g., memory 1006, processor 1004, communication interface 1008).

[0096] The processor 1004 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0097] The memory 1006 may include volatile memory, such as random access memory (RAM). The processor 1004 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0098] The memory 1006 stores executable program code, and the processor 1004 executes the executable program code to implement the aforementioned functions respectively. Figure 9 The receiving module 901 and processing module 902 shown herein perform their functions to implement the data compliance detection method described in the above embodiments. That is, the memory 1006 stores instructions for executing the data compliance detection method described in the above embodiments.

[0099] Alternatively, the memory 1006 stores executable code, and the processor 1004 executes the executable code to implement the aforementioned functions respectively. Figure 9 The data compliance detection device 900 shown in the diagram performs the functions of the data compliance detection method described in the above embodiments. That is, the memory 1006 stores instructions for executing the data compliance detection method described in the above embodiments.

[0100] The communication interface 1008 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 1000 and other devices or communication networks.

[0101] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0102] like Figure 11 As shown, the computing device cluster includes at least one computing device 1000. The memory 1006 in one or more computing devices 1000 in the computing device cluster may store the same instructions for executing the data compliance detection method described in the above embodiments.

[0103] In some possible implementations, the memory 1006 of one or more computing devices 1000 in the computing device cluster may also store partial instructions for executing the data compliance detection method described in the above embodiments. In other words, a combination of one or more computing devices 1000 can jointly execute instructions for executing the data compliance detection method described in the above embodiments.

[0104] It should be noted that the memory 1006 in different computing devices 1000 within the computing device cluster can store different instructions, each used to execute the aforementioned instructions. Figure 9 This describes some of the functions of the data compliance detection device 900 shown. Specifically, the instructions stored in the memory 1006 of different computing devices 1000 can implement the functions of one or more modules in the receiving module 901 and processing module 902.

[0105] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 12 One possible implementation is shown. For example... Figure 12 As shown, two computing devices 1000A and 1000B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 1006 in computing device 1000A stores instructions for performing the functions of the receiving module 901. Simultaneously, the memory 1006 in computing device 1000B stores instructions for performing the functions of the processing module 902.

[0106] It should be understood that Figure 12The functions of computing device 1000A shown can also be performed by multiple computing devices 1000. Similarly, the functions of computing device 1000B can also be performed by multiple computing devices 1000.

[0107] This application also provides another computing device cluster. The connection relationships between the computing devices in this computing device cluster can be similarly referred to... Figure 11 and Figure 12 The connection method of the computing device cluster is different in that the memory 1006 of one or more computing devices 1000 in the computing device cluster can store the same instructions for executing the methods in the above embodiments.

[0108] In some possible implementations, the memory 1006 of one or more computing devices 1000 in the computing device cluster may also store partial instructions for executing the aforementioned data compliance detection method. In other words, a combination of one or more computing devices 1000 can jointly execute the instructions for executing the aforementioned data compliance detection method.

[0109] It should be understood that each step of the above method embodiments can be accomplished by hardware logic circuits or software instructions in a processor.

[0110] Based on the methods in the above embodiments, this application provides a computer-readable storage medium including computer program instructions. When executed by a cluster of computing devices including at least one computing device, the computer program instructions cause the cluster of computing devices to perform the methods in the above embodiments. Exemplarily, the computer-readable storage medium can be any available medium that the computing device can store, or a data storage device such as a data center containing one or more available media. Available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives).

[0111] Based on the methods in the above embodiments, this application provides a computer program product containing instructions that, when executed by a cluster of computing devices containing at least one computing device, cause the cluster of computing devices to perform the methods in the above embodiments.

[0112] Based on the methods described in the above embodiments, this application also provides a data compliance detection device. Please refer to... Figure 13 , Figure 13 This is a schematic diagram of another data compliance detection device provided in an embodiment of this application. Figure 13As shown, the data compliance detection device 1300 includes one or more processors 1301 and interface circuits 1302. Optionally, the data compliance detection device 1300 may also include a bus 1303. Wherein:

[0113] Processor 1301 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed through integrated logic circuits in the hardware of processor 1301 or through software instructions. Processor 1301 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods and steps disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor.

[0114] The interface circuit 1302 can be used to send or receive data, instructions or information. The processor 1301 can use the data, instructions or other information received by the interface circuit 1302 to process the data, instructions or other information, and can send the processed information out through the interface circuit 1302.

[0115] Optionally, the data compliance detection device 1300 also includes a memory, which may include read-only memory and random access memory, and provides operating instructions and data to the processor. A portion of the memory may also include non-volatile random access memory (NVRAM).

[0116] Optionally, the memory stores executable software modules or data structures, and the processor can execute corresponding operations by calling the operation instructions stored in the memory (which may be stored in the operating system).

[0117] Optionally, the interface circuit 1302 can be used to output the execution results of the processor 1301.

[0118] It should be noted that the functions of the processor 1301 and the interface circuit 1302 can be implemented through hardware design, software design, or a combination of hardware and software; no restrictions are imposed here.

[0119] It should be understood that each step of the above method embodiments can be completed by hardware logic circuits or software instructions in a processor.

[0120] It is understood that the processor in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0121] The method steps in the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0122] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0123] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application.

[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. A data compliance detection method, characterized in that, The method includes: Receive queries from users; Based on the similarity between the query question and the content in the sample library, it is determined whether the query question is compliant. The sample library contains user-reported missed detection examples and false positive examples.

2. The method according to claim 1, characterized in that, The process of determining whether a query is compliant based on its similarity to content in the sample database includes: If the query question is similar to the missed sample, the query question is determined to be non-compliant. If the query question is similar to the false positive example, the query question is determined to be compliant.

3. The method according to claim 1 or 2, characterized in that, The process of determining whether a query is compliant based on its similarity to content in the sample database includes: The content in the sample library is used as system prompt words, and the query question is used as user prompt words. These are input into a neural network model to determine whether the query question is compliant.

4. The method according to claim 4, characterized in that, The method further includes: Cache and compress the processing results of each network layer in the neural network model on the content in the sample library; Alternatively, the pre-stored processing results of each network layer in the neural network model on the contents of the sample library can be loaded.

5. The method according to any one of claims 1-4, characterized in that, The sample library includes a black sample library and a white sample library. The black sample library contains the missed detection samples, and the white sample library contains the false positive samples. Before determining whether a query is compliant based on its similarity to content in the sample database, the following steps are also included: Cluster the contents of the black sample library to obtain N clusters, and cluster the contents of the white sample library to obtain M clusters, where N and M are both greater than or equal to 1; Semantic compression is performed on the content of each of the N clusters, and semantic compression is performed on the content of each of the M clusters, wherein the semantic compression is used to reduce duplicate or similar content.

6. A data compliance detection device, characterized in that, include: The receiving module is used to receive queries from users; The processing module is used to determine whether the query question is compliant based on the similarity between the query question and the content in the sample library, wherein the sample library contains missed detection samples and false positive samples reported by users.

7. The apparatus according to claim 6, characterized in that, When determining whether a query is compliant based on the similarity between the query question and the content in the sample library, the processing module is specifically used for: If the query question is similar to the missed sample, the query question is determined to be non-compliant. If the query question is similar to the false positive example, the query question is determined to be compliant.

8. The apparatus according to claim 6 or 7, characterized in that, When determining whether a query is compliant based on the similarity between the query question and the content in the sample library, the processing module is specifically used for: The content in the sample library is used as system prompt words, and the query question is used as user prompt words. These are input into a neural network model to determine whether the query question is compliant.

9. The apparatus according to claim 8, characterized in that, The processing module is further configured to: Cache and compress the processing results of each network layer in the neural network model on the content in the sample library; Alternatively, the pre-stored processing results of each network layer in the neural network model on the contents of the sample library can be loaded.

10. The apparatus according to any one of claims 6-9, characterized in that, The sample library includes a black sample library and a white sample library. The black sample library contains the missed detection samples, and the white sample library contains the false positive samples. Before determining whether a query is compliant based on its similarity to content in the sample database, the processing module further performs the following: Cluster the contents of the black sample library to obtain N clusters, and cluster the contents of the white sample library to obtain M clusters, where N and M are both greater than or equal to 1; Semantic compression is performed on the content of each of the N clusters, and semantic compression is performed on the content of each of the M clusters, wherein the semantic compression is used to reduce duplicate or similar content.

11. A data compliance detection device, characterized in that, Includes at least one processor and interface; The at least one processor obtains program instructions through the interface; The at least one processor is configured to execute the program line instructions to implement the method as described in any one of claims 1-5.

12. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1-5.

13. A computer-readable storage medium, characterized in that, The method includes computer program instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method as described in any one of claims 1-5, wherein the cluster of computing devices includes at least one computing device.

14. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster causes the computing device cluster to perform the method as described in any one of claims 1-5, wherein the computing device cluster includes at least one computing device.

Images