Data screening method, device and electronic equipment

Abstract

The application discloses a data screening method and device and electronic equipment. It is related to the field of data processing. The method comprises the following steps: determining the data type and application scenario of the to-be-processed data; determining the target detection strategy of the to-be-processed data according to the application scenario, and detecting the to-be-processed data according to the target detection strategy to obtain a first detection result; in the case that the first detection result represents that the to-be-processed data is normal, obtaining the detection items associated with the to-be-processed data from a data detection table according to the application scenario and the data type to obtain a target detection item set; inputting the target detection item set and the to-be-processed data into a preset large language model set to obtain a second detection result, and determining the screening operation performed on the to-be-processed data according to the second detection result. Through the application, the problem that the accuracy of the data screening operation is low due to the language level detection and screening of the data by only relying on the text quality index in the related art is solved.

Description

Technical Field

[0001] This application relates to the field of data processing, and more specifically, to a data filtering method, apparatus, and electronic device. Background Technology

[0002] In the field of sports and fitness, as the demand for personalized fitness guidance continues to increase, more and more institutions are using large language models to provide users with personalized fitness guidance services. By leveraging the powerful natural language understanding and generation capabilities of large language models, they can provide users with real-time, personalized professional services.

[0003] When training large language models in sports-related domains, the training data often contains a large amount of professional content in the sports field, such as training plans, exercise guidance, and health advice. The accuracy of this data directly affects the accuracy of the model's output. Therefore, before training the model, it is necessary to perform quality checks on the training data and filter out data that does not meet the quality standards to ensure the accuracy of the model training.

[0004] Current data quality assessments primarily rely on general text quality metrics for language-related quality checks, such as fluency, repetition rate, and toxicity testing. These checks only focus on surface-level language forms and cannot identify critical errors specific to sports science, such as excessively high repetitions in a single set, lack of warm-up or cool-down phases, or movement parameters that violate physiological limits. As a result, a large amount of data with safety risks or errors is not effectively screened out, affecting the accuracy of the training data content and ultimately impacting the accuracy of the model training results.

[0005] There is currently no effective solution to the problem that the accuracy of data filtering operations is low because the technology relies solely on text quality indicators for language-level detection and filtering. Summary of the Invention

[0006] The main objective of this application is to provide a data filtering method, apparatus, and electronic device to solve the problem that the accuracy of data filtering operations is low due to relying solely on text quality indicators for language-level detection and filtering in related technologies.

[0007] To achieve the above objectives, according to one aspect of this application, a data filtering method is provided. The method includes: retrieving data to be processed from a database, and determining the data type and application scenario of the data to be processed, wherein the data to be processed is single-turn dialogue data or multi-turn dialogue data; determining a target detection strategy for the data to be processed based on the application scenario, and detecting the data to be processed according to the target detection strategy to obtain a first detection result; if the first detection result indicates that the data to be processed has no anomalies, retrieving detection items associated with the data to be processed from a data detection table according to the application scenario and data type to obtain a target detection item set; inputting the target detection item set and the data to be processed into a preset large language model set to obtain a second detection result, and determining the filtering operation to be performed on the data to be processed based on the second detection result.

[0008] Optionally, determining the target detection strategy for the data to be processed based on the application scenario includes: obtaining a detection strategy associated with the application scenario from a strategy library to obtain a first detection strategy; obtaining the data content of the data to be processed and determining the feature information of the data to be processed based on the data content; determining the feature type to which the feature information belongs and selecting a second detection strategy from the first detection strategy based on the feature type, wherein the second detection strategy contains the detection content corresponding to the feature type; and determining the second detection strategy as the target detection strategy.

[0009] Optionally, the target detection strategy includes M sub-detection strategies. Detecting the data to be processed according to each target detection strategy to obtain a first detection result includes: detecting the data to be processed according to each sub-detection strategy to obtain M first sub-detection results, where M is a positive integer; determining whether there is a target sub-detection result with an anomaly level of 1 among the M first sub-detection results; if a target sub-detection result exists among the M first sub-detection results, determining that the first detection result is anomaly present; if no target sub-detection result exists among the M first sub-detection results, determining that the first detection result is anomaly absent.

[0010] Optionally, the method further includes: if the first detection result indicates that the data to be processed is abnormal, determining the filtering operation to be performed on the data to be processed as deleting the data to be processed from the database.

[0011] Optionally, based on the application scenario and data type, the detection items associated with the data to be processed are obtained from the data detection table to obtain the target detection item set, which includes: obtaining general detection items from the data detection table and obtaining scenario detection items associated with the application scenario; obtaining a set of condition information associated with the data detection table and determining the target condition information that the data to be processed meets; obtaining condition detection items corresponding to the target condition information from the data detection table and obtaining round detection items associated with the data type; and constructing the target detection item set based on the general detection items, scenario detection items, condition detection items, and round detection items.

[0012] Optionally, the preset large language model set includes N preset large language models. Inputting the target detection item set and the data to be processed into the preset large language model set to obtain the second detection result includes: for any preset large language model, inputting the target detection item set and the data to be processed into the preset large language model respectively to obtain a second sub-detection result. The second sub-detection result includes a detection score set, which includes the detection scores of the preset large language model for each target detection item in the target detection item set; determining whether there are any scores less than or equal to a target threshold in the detection score set of the second sub-detection result; if there are any scores less than or equal to the target threshold in the detection score set, determining that the second sub-detection result is abnormal; if there are no scores less than or equal to the target threshold in the detection score set, determining that the second sub-detection result is not abnormal; and combining the second sub-detection results of each preset large language model to obtain the second detection result.

[0013] Optionally, determining the filtering operation to be performed on the data to be processed based on the second detection result includes: determining whether there is a second sub-detection result with an abnormal characterization in the second detection result; if there is a second sub-detection result with an abnormal characterization in the second detection result, determining the filtering operation to be performed on the data to be processed to delete the data to be processed from the database; if there is no second sub-detection result with an abnormal characterization in the second detection result, determining the filtering operation to be performed on the data to be processed to retain the data to be processed in the database.

[0014] To achieve the above objectives, according to another aspect of this application, a data filtering apparatus is provided. The apparatus includes: a determining unit, configured to retrieve data to be processed from a database and determine the data type and application scenario of the data to be processed, wherein the data to be processed is single-turn dialogue data or multi-turn dialogue data; a detecting unit, configured to determine a target detection strategy for the data to be processed based on the application scenario, and to detect the data to be processed according to the target detection strategy to obtain a first detection result; an acquiring unit, configured to, when the first detection result indicates that the data to be processed is normal, retrieve detection items associated with the data to be processed from a data detection table based on the application scenario and data type to obtain a target detection item set; and a processing unit, configured to input the target detection item set and the data to be processed into a preset large language model set to obtain a second detection result, and to determine the filtering operation to be performed on the data to be processed based on the second detection result.

[0015] To achieve the above objectives, according to another aspect of this application, an electronic device is provided, the electronic device including a memory storing an executable program; and a processor for running the program, wherein the program executes the above-described data filtering method during runtime.

[0016] To achieve the above objectives, according to another aspect of this application, a computer program product is provided, including computer instructions that, when executed by a processor, implement the steps of the data filtering method described above.

[0017] In this embodiment, the process involves retrieving data to be processed from a database and determining the data type and application scenario. The data to be processed is either single-turn or multi-turn dialogue data. A target detection strategy is determined based on the application scenario, and the data is then detected according to the target detection strategy to obtain a first detection result. If the first detection result indicates that the data to be processed is normal, detection items associated with the data to be processed are retrieved from a data detection table based on the application scenario and data type to obtain a target detection item set. The target detection item set and the data to be processed are input into a preset large language model set to obtain a second detection result. Based on the second detection result, the method for performing a filtering operation on the data to be processed is determined. By determining the target detection strategy based on the data type and application scenario, the data to be processed is detected once. Furthermore, by determining the detection items based on the application scenario and data type, and using these detection items to perform a second detection on the data to be processed, it is determined whether the data to be processed needs to be filtered. This achieves the technical effect of improving the accuracy of data quality detection and filtering, thereby solving the technical problem in related technologies where the accuracy of data filtering operations is low due to relying solely on text quality indicators for language-level detection and filtering. Attached Figure Description

[0018] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0019] Figure 1 A hardware structure block diagram of a computer terminal for implementing a data filtering method is shown.

[0020] Figure 2 This is a flowchart of the data filtering method provided in Embodiment 1 of this application;

[0021] Figure 3 This is a schematic diagram of the data filtering device provided according to Embodiment 2 of this application;

[0022] Figure 4 This is a structural block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0023] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0024] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0026] It should be noted that the data filtering methods, apparatus, and electronic equipment defined in this disclosure can be used in the field of data processing, or in any field other than data processing. The application fields of the data filtering methods, apparatus, and electronic equipment defined in this disclosure are not limited.

[0027] It should be noted that all information, user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, and displayed data) used in this application are information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with the relevant regulations and standards of the relevant regions, have taken necessary measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse use. If the user chooses to refuse, the process proceeds to the expert decision-making process. For example, this system has interfaces with relevant users or organizations. Before obtaining relevant information, a request to obtain the information needs to be sent to the aforementioned user or organization through the interface. After receiving consent from the aforementioned user or organization, the relevant information is obtained. Users can view the purpose of data use in real time through the authorization interface and have the right to withdraw authorization or delete data at any time. After authorization is withdrawn, the system will terminate the relevant data processing within 24 hours.

[0028] The embodiments or examples disclosed herein are not exhaustive, but merely illustrative of some embodiments or examples, and are not intended to limit the scope of protection of this disclosure. Unless otherwise specified, each step in a particular embodiment or example can be implemented as an independent embodiment, and the steps can be arbitrarily combined. For example, a solution after removing some steps in a particular embodiment or example can also be implemented as an independent embodiment, and the order of the steps in a particular embodiment or example can be arbitrarily interchanged. Furthermore, optional methods or examples in a particular embodiment or example can be arbitrarily combined; moreover, embodiments or examples can be arbitrarily combined. For example, some or all steps of different embodiments or examples can be arbitrarily combined, and a particular embodiment or example can be arbitrarily combined with optional methods or examples of other embodiments or examples.

[0029] Example 1

[0030] According to an embodiment of this application, an embodiment of a data filtering method is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Also, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0031] The method embodiment provided in Embodiment 1 of this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware block diagram of a computer terminal for implementing a data filtering method is shown. Figure 1 As shown, the computer terminal 10 (or mobile device) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, processing devices such as microprocessors or programmable logic devices), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface, a universal serial bus port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0032] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0033] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the data filtering method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the above-mentioned data filtering method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0034] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0035] The display may be, for example, a touchscreen LCD display that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0036] Under the aforementioned operating environment, this application provides the following: Figure 2 The data filtering method shown. Figure 2 This is a flowchart of the data filtering method provided in Embodiment 1 of this application, such as... Figure 2 As shown, the method includes:

[0037] Step S201: Obtain the data to be processed from the database and determine the data type and application scenario of the data to be processed, wherein the data to be processed is single-turn dialogue data or multi-turn dialogue data.

[0038] It should be noted that the execution entity in this embodiment can be a data filtering system. This system can perform quality detection and filtering operations on the training data of the large language model that has completed its initial construction, thereby improving the accuracy of the training data and thus improving the model training effect.

[0039] It should be noted that "database" refers to a structured or semi-structured data source storing training data, "data to be processed" refers to training data that requires quality assessment, and can be single-turn or multi-turn dialogue data. The application scenario for the data to be processed is the sports and health field; that is, the data to be processed is dialogue data within the sports and health field, and its content is related to the sports and health field. "Data type" refers to the data's structural form, including single-turn and multi-turn dialogue data. "Application scenario" refers to the specific usage environment the data is intended for, such as training plan generation, health consultation, and multi-turn rehabilitation interaction. Single-turn dialogue data contains a set of question-and-answer pairs, while multi-turn dialogue data contains multiple sets of consecutive question-and-answer pairs.

[0040] For example, before training a large language model using training data, the training data needs to be quality checked. First, each training data point needs to be retrieved from the database storing the training data and treated as data to be processed. Based on the structural features and metadata fields of the data to be processed, the system automatically identifies whether the data to be processed is single-turn dialogue data or multi-turn dialogue data, thereby determining the data type of the data to be processed. Single-turn dialogue data is represented as an independent pair of user input and model response, and its context information is limited to the current turn. Multi-turn dialogue data consists of multiple consecutive user-model interaction turns, with obvious semantic dependencies and state continuity.

[0041] Similarly, the label fields of the data to be processed can be read and combined with its semantic patterns to determine its corresponding application scenario, such as training plan design and dietary suggestion generation. This recognition process can complete the classification based on preset field mapping rules, keyword matching patterns, and structural templates. For example, data containing fields such as number of sets, repetitions, rest time, and equipment usage can be classified as a training plan design scenario.

[0042] Step S202: Determine the target detection strategy for the data to be processed based on the application scenario, and perform detection on the data to be processed according to the target detection strategy to obtain the first detection result.

[0043] It should be noted that the target detection strategy refers to a set of preliminary detection rules predefined for a specific application scenario, based on rules or lightweight models. This detection strategy may include multiple preset verification rules, as well as the verification requirements and verification process for each verification rule. The data to be processed can be verified according to each verification rule in the target detection strategy to determine the first detection result of the data to be processed. The first detection result can be used to characterize the anomaly level of the data to be processed, such as high severity anomaly, medium severity anomaly, and low severity anomaly.

[0044] For example, after determining the application scenario of the data to be processed, a matching target detection strategy can be called from the preset strategy library. The target detection strategy can include multiple verification rules. For example, in the training plan design scenario, the target detection strategy includes the detection of rules such as insufficient title length, zero paragraphs, negative action count, and total duration exceeding a reasonable range. The verification rules in the target detection strategy are used to detect the data to be processed, thereby determining whether there are any abnormalities in the data to be processed.

[0045] If the data to be processed matches any rule, the first detection result is that there is an anomaly, and the data to be processed can be removed and will not enter the subsequent process; if the data to be processed does not match any rule, the first detection result is that there is no anomaly, and the data continues to execute the subsequent process, thus completing one detection operation on the data to be tested.

[0046] For example, when the data to be tested is fitness training plan data, it can be validated according to the following six categories of validation rules:

[0047] Category 1: Metadata validation, used to check the completeness and reasonableness of the top-level fields of the training plan: for example, rule 1.2: insufficient title length (<4 characters) is identified as a medium severity anomaly;

[0048] Category 2: Paragraph-level structure verification, used to check the structural rationality of each paragraph in the training plan: for example, rule 2.3: containing only 1 paragraph (lacking training structure) is identified as a high-severity anomaly;

[0049] Category 3: Action-level parameter verification, used to check whether the parameters of each action are within the biomechanically reasonable range: for example, rule 3.5: action duration ≤ 0, identified as moderate to severe abnormality;

[0050] Category 4: Time Consistency Validation, used to detect the consistency between computation time and declaration time: for example: Rule 4.1: A ratio of computation time to declaration time > 2.0 or < 0.3 is identified as a high severity anomaly;

[0051] Category 5: Statistical indicator verification, used to detect the overall statistical characteristics of the training plan: for example, rule 5.1: total number of actions ≤ 2, identified as a high severity anomaly;

[0052] Category 6: Structural integrity verification, used to check the rationality of the structure of the training plan: for example, rule 6.1: missing warm-up paragraph is identified as a moderate to severe anomaly.

[0053] Step S203: If the first detection result indicates that the data to be processed is not abnormal, the detection items associated with the data to be processed are obtained from the data detection table according to the application scenario and data type, thus obtaining the target detection item set.

[0054] It should be noted that the data detection table is a structured indexed database that stores the mapping relationship between application scenarios, data types and detection items; the target detection item set refers to the set of evaluation dimensions that are dynamically extracted based on this mapping relationship and match the data to be processed.

[0055] For example, if the first test result is normal, it is also necessary to query a preset data test table based on the application scenario and data type of the data. This table has a two-dimensional mapping structure, with rows corresponding to combinations of application scenarios and data types, and columns corresponding to enabled test items. Each test item is an assessable quality dimension, such as "course accuracy" or "course comprehensibility".

[0056] Furthermore, based on the labels of the data to be processed, the corresponding rows are matched, and all associated detection items under that row are extracted to form a target detection item set. Each item in this set has clearly specified evaluation content and scoring criteria, serving as the input instruction framework for subsequent large language model evaluation.

[0057] Step S204: Input the target detection item set and the data to be processed into the preset large language model set to obtain the second detection result, and determine the filtering operation to be performed on the data to be processed based on the second detection result.

[0058] It should be noted that the pre-set large language model set refers to a set of models composed of multiple large language models with different architectures and heterogeneous training methods.

[0059] For example, after obtaining the set of object detection items, the set of object detection items and the data to be processed can be simultaneously input into a pre-defined large language model set. This set can consist of at least four models with significantly different architectures, whose training methods, language preferences, and security judgment mechanisms complement each other. Each model independently receives the same input and makes judgments item by item according to each evaluation dimension defined in the set of object detection items, outputting a structured score for each detection item, thus obtaining a score set output by each model. Based on the score set output by each model, a second detection result is obtained, and based on the second detection result, it is determined whether the data to be processed needs to be filtered.

[0060] For example, the average value of the score set output by each model can be calculated, and it can be determined whether each average value is greater than a preset threshold. If each average value is greater than the preset threshold, the second detection result is determined to be a pass detection.

[0061] Therefore, this embodiment obtains the data to be processed from the database and identifies its type and scenario, drives the target detection strategy to perform preliminary filtering, then dynamically extracts the target detection item set according to the scenario and type, and finally performs accurate and multi-dimensional collaborative evaluation by a heterogeneous large language model set, forming a complete screening decision closed loop.

[0062] The data filtering method provided in this application embodiment improves the accuracy of data quality detection and filtering by obtaining data to be processed from a database and determining the data type and application scenario, wherein the data to be processed is single-turn or multi-turn dialogue data; determining a target detection strategy for the data to be processed based on the application scenario, and detecting the data to be processed according to the target detection strategy to obtain a first detection result; if the first detection result indicates that the data to be processed is normal, obtaining detection items associated with the data to be processed from a data detection table according to the application scenario and data type to obtain a target detection item set; inputting the target detection item set and the data to be processed into a preset large language model set to obtain a second detection result, and determining the filtering operation to be performed on the data to be processed based on the second detection result. By determining the target detection strategy based on the data type and application scenario, the data to be processed is detected once, and the detection items are determined based on the application scenario and data type, and the detection items are used to detect the data to be processed a second time, thereby determining whether the data to be processed needs to be filtered. This achieves the technical effect of improving the accuracy of data quality detection and filtering, and solves the technical problem in related technologies where the accuracy of data filtering operations is low due to relying solely on text quality indicators for language-level detection and filtering.

[0063] Optionally, in the data filtering method provided in this application embodiment, determining the target detection strategy for the data to be processed according to the application scenario includes: obtaining a detection strategy associated with the application scenario from a strategy library to obtain a first detection strategy; obtaining the data content of the data to be processed and determining the feature information of the data to be processed based on the data content; determining the feature type to which the feature information belongs, and selecting a second detection strategy from the first detection strategy based on the feature type, wherein the second detection strategy contains detection content corresponding to the feature type; and determining the second detection strategy as the target detection strategy.

[0064] It should be noted that the strategy library is a structured database that stores various detection strategies, each of which is bound to a specific application scenario; the first detection strategy refers to the set of basic detection strategies that are initially matched from the strategy library and associated with the application scenario; the data content refers to the textual semantic and structural information contained in the data to be processed; the feature information refers to the key identifiers extracted from the data content that can characterize the inherent structure or semantic attributes of the data; and the feature type refers to the preset category to which the feature information is classified, such as the number of movements or the duration of movement.

[0065] For example, when determining the target detection strategy, the initial detection strategy is first retrieved from the preset strategy library and loaded according to the application scenario of the data to be processed, forming the first detection strategy. The first detection strategy is a generalized set of rules that covers all possible quality detection items in the application scenario, but its coverage is too broad and it is not optimized for specific data features.

[0066] Furthermore, the system performs structured parsing of the data content to be processed, extracting key identifiers at the semantic and structural levels to form feature information. The extraction of this feature information relies on preset semantic pattern recognition rules, such as identifying the existence of parameter fields such as "number of times," "number of sets," and "rest time," and determining whether the paragraph contains the three structural modules of warm-up, main body, and relaxation.

[0067] After obtaining feature information, the system categorizes each extracted feature into a preset feature type according to a predefined feature classification system. For example, if the identified content includes "running," the feature type can be determined as "sports"; if the identified content includes "fatigue relief," the feature type can be determined as "health knowledge," etc. Each feature type has a corresponding sub-strategy in the first detection strategy. Based on the feature type it belongs to, the system extracts directly related detection content from the first detection strategy, thus forming the second detection strategy. The second detection strategy is a subset of the first detection strategy, retaining only detection rules that are strongly correlated with the feature type of the current data, and its content is highly focused and targeted.

[0068] Ultimately, the system identifies the second detection strategy as the target detection strategy and passes it on to subsequent detection processes.

[0069] This embodiment achieves an upgrade from general matching to precise adaptation of detection strategies by screening detection strategies, avoiding redundant execution of irrelevant detection items during the evaluation process, and improving the detection efficiency and accuracy of the data to be processed.

[0070] Optionally, in the data filtering method provided in this application embodiment, the target detection strategy includes M sub-detection strategies. Detecting the data to be processed according to the target detection strategy to obtain a first detection result includes: detecting the data to be processed according to each sub-detection strategy to obtain M first sub-detection results, where M is a positive integer; determining whether there is a target sub-detection result with an anomaly level of the first among the M first sub-detection results; if there is a target sub-detection result among the M first sub-detection results, determining that the first detection result is abnormal; if there is no target sub-detection result among the M first sub-detection results, determining that the first detection result is not abnormal.

[0071] It should be noted that a sub-detection strategy refers to an independently defined single detection rule unit contained within an object detection strategy.

[0072] For example, when detecting the data to be processed, the M sub-detection strategies contained in the target detection strategy are first loaded into the detection engine one by one, so that each sub-detection strategy has an independent execution context and input data permissions.

[0073] Furthermore, for each piece of data to be processed, each sub-detection strategy is called in sequence to perform independent rule matching and logical judgment, generating M first sub-detection results. Each first sub-detection result is a structured judgment item, which includes the detection item number, judgment conclusion and anomaly level. The anomaly level is divided into first level, second level and third level, where the first level represents high severity anomaly, the second level represents medium severity anomaly and the third level represents low severity anomaly.

[0074] After obtaining M first sub-detection results, a global scan is performed on each of them to check for any target sub-detection results with an anomaly level of 1. If at least one sub-detection result is determined to have an anomaly level of 1, the system immediately terminates the subsequent evaluation, uniformly classifying the first detection results as having anomalies. This data will be directly discarded and will not proceed to the subsequent large language model evaluation stage. If all anomaly levels among the M first sub-detection results are lower than 1, meaning only non-core defects of 2nd or 3rd level exist, the system determines that the first detection results do not have anomalies, and the data proceeds to the next stage of the large language model evaluation.

[0075] This embodiment introduces an anomaly level priority determination mechanism to intercept high-severity anomalies, thereby completing one detection of the data to be processed and improving detection efficiency.

[0076] Optionally, in the data filtering method provided in the embodiments of this application, the method further includes: when the first detection result indicates that the data to be processed is abnormal, determining the filtering operation to be performed on the data to be processed as deleting the data to be processed from the database.

[0077] For example, if a high-severity anomaly is found in the first detection result, the first detection result is characterized as having an anomaly. In this case, the data to be processed needs to be deleted from the training set, thereby improving the accuracy and availability of the data in the training set.

[0078] Optionally, in the data filtering method provided in this application embodiment, obtaining the target detection item set by retrieving detection items associated with the data to be processed from the data detection table according to the application scenario and data type includes: retrieving general detection items from the data detection table and retrieving scenario detection items associated with the application scenario; retrieving the condition information set associated with the data detection table and determining the target condition information that the data to be processed meets; retrieving the condition detection items corresponding to the target condition information from the data detection table and retrieving the round detection items associated with the data type; and constructing the target detection item set according to the general detection items, scenario detection items, condition detection items, and round detection items.

[0079] It should be noted that the data detection table is a pre-built structured mapping database that stores the multi-dimensional relationships between application scenarios, data types, conditional information, and detection items; general detection items refer to basic and unconditionally dependent quality assessment dimensions applicable to all application scenarios and data types; scenario detection items refer to detection items directly bound to the application scenario of the data to be processed, reflecting the quality requirements specific to that functional domain; the conditional information set is the set of prerequisite conditions defined in the data detection table used to trigger specific detection items; target conditional information refers to one or more specific conditions from the conditional information set that the data to be processed satisfies in terms of content or structure; conditional detection items refer to detection dimensions that should only be activated when the target conditional information is true; round detection items refer to detection dimensions designed for the interaction structure features corresponding to whether the data to be processed is single-round or multi-round dialogue data; the target detection item set is the final complete set of evaluation dimensions composed of the four types of detection items: general, scenario, conditional, and round, used to guide the subsequent evaluation of the large language model.

[0080] For example, when determining the target set of detection items for the data to be processed, it is first necessary to extract general detection items from the data detection table. These items are basic quality requirements that all data types and application scenarios in the health and fitness field must meet, such as "clarity of language expression" and "consistency in terminology use", which form the basis of the evaluation.

[0081] Furthermore, based on the application scenario of the data to be processed, the relevant scenario detection items are retrieved from the data detection table. These items are specific to a particular task type, such as "forced existence of warm-up segments" and "reasonable range of motion density" in the "training plan generation" scenario.

[0082] Similarly, the semantic content of the data to be processed also needs to be parsed and compared with the predefined set of condition information in the data detection table to identify whether the current data meets a certain trigger condition, such as "whether there is a follow-up question" or "whether a modification request has been made". Any data that meets the condition is identified as the target condition information. For each target condition information, the system extracts its corresponding condition detection item from the data detection table. This type of item is a conditional dimension and only participates in the evaluation when the condition is met. For example, "fatigue management" is only enabled when there are multiple training plans, and "correction response quality" only takes effect when the user corrects an error.

[0083] Similarly, it is also necessary to extract the matching round detection items based on the data type of the data to be processed (single-turn or multi-turn dialogue), that is, the unique detection items for single-turn dialogue data and multi-turn dialogue data.

[0084] Finally, the system aggregates general detection items, scene detection items, condition detection items, and round detection items, removes redundant items, and constructs a unique, complete, and conflict-free set of target detection items.

[0085] It should be noted that security detection items can also be configured to detect the data to be processed from a security perspective.

[0086] This embodiment achieves the technical effect of accurately and completely obtaining the target detection item set of the data to be processed by acquiring the detection items from various dimensions, thereby improving the accuracy of subsequent detection of the data to be processed using a large language model.

[0087] Optionally, in the data filtering method provided in this application embodiment, the preset large language model set includes N preset large language models. Inputting the target detection item set and the data to be processed into the preset large language model set to obtain the second detection result includes: for any preset large language model, inputting the target detection item set and the data to be processed into the preset large language model respectively to obtain a second sub-detection result, wherein the second sub-detection result includes a detection score set, which includes the detection scores of the preset large language model for each target detection item in the target detection item set; determining whether there is a score less than or equal to a target threshold in the detection score set of the second sub-detection result; if there is a score less than or equal to the target threshold in the detection score set, determining that the second sub-detection result is abnormal; if there is no score less than or equal to the target threshold in the detection score set, determining that the second sub-detection result is not abnormal; combining the second sub-detection results of each preset large language model to obtain the second detection result.

[0088] It should be noted that the pre-set large language model set refers to a collection of large language models with heterogeneous architectures, different training objectives, and different language preferences. For example, the pre-set large language model set may include: Model A, which has specific advantages in safety and ethical judgment; Model B, which has specific advantages in technical accuracy and instruction compliance; Model C, which has specific advantages in cross-domain knowledge integration; and Model D, which has specific advantages in understanding Chinese fitness terminology and cultural context. The detection score set is the set of quantitative scores contained in the second sub-detection results, corresponding to each item in the target detection item set, with a score range of 1 to 3 points.

[0089] For example, when determining the second detection result, it is necessary to use each of the large language models in the preset large language model set to perform a detection on the data to be processed, obtain multiple second sub-detection results, and analyze the multiple second sub-detection results to determine the final second detection result.

[0090] For each pre-defined large language model in the pre-defined large language model set, the target detection item set and the data to be processed need to be injected into the inference interface of each model, respectively. Without sharing intermediate states or referring to the output of other models, each pre-defined large language model independently generates a corresponding detection score based on each evaluation dimension in the target detection item set, forming a detection score set, where each score in the set represents the model's quality judgment for that dimension.

[0091] After obtaining the set of detection scores output by each model, each set of detection scores needs to be analyzed to determine whether any score is less than or equal to the target threshold. If at least one score is less than or equal to the target threshold, the model determines that the data to be processed has not met the basic quality requirements in at least one core dimension, and therefore marks the second sub-detection result of the model as having an anomaly; if all scores in the detection score set are greater than the target threshold, it indicates that the model considers all evaluation dimensions to have reached a qualified or higher level, and the second sub-detection result is marked as having no anomalies.

[0092] The system sequentially executes the above process on N preset large language models to obtain N second sub-detection results, and then performs a structured combination according to the model number to form a complete second detection result.

[0093] This embodiment uses multiple large language models to detect the data to be processed, achieving the technical effect of accurately determining the detection results.

[0094] Optionally, in the data filtering method provided in this application embodiment, determining the filtering operation to be performed on the data to be processed based on the second detection result includes: determining whether there is a second sub-detection result with an abnormal characterization in the second detection result; if there is a second sub-detection result with an abnormal characterization in the second detection result, determining the filtering operation to be performed on the data to be processed to delete the data to be processed from the database; if there is no second sub-detection result with an abnormal characterization in the second detection result, determining the filtering operation to be performed on the data to be processed to retain the data to be processed in the database.

[0095] For example, when determining whether to delete data to be processed, the system first performs an integrity check on the second detection result to confirm that it contains the second sub-detection results of all N preset large language models. Subsequently, the system iterates through each second sub-detection result in the second detection result and checks item by item whether there is at least one second sub-detection result marked as having an anomaly.

[0096] If any of the preset large language models outputs an anomaly, the data to be processed is considered to have failed to meet the minimum quality standard in a certain evaluation dimension by at least one professional model, regardless of whether other models pass or fail. In this case, the system immediately triggers a filtering operation: deleting the data from the database. After deletion, the system generates an audit log, recording the deletion time, the data's unique identifier, the model number that triggered the anomaly, and the corresponding anomaly dimension, ensuring full traceability.

[0097] If all second sub-detection results in the second detection result are found to be free of anomalies, meaning that all preset large language models have determined that the data to be processed has reached the qualified or above level in all target detection items, the system will determine that the data meets the admission criteria for high-quality training corpus, and will determine the filtering operation performed on the data to be processed to be retained in the database, so that it can enter the candidate pool of the subsequent training dataset for use in the subsequent model training stage.

[0098] This embodiment uses consensus voting to determine whether data to be processed needs to be deleted, thereby achieving accurate quality inspection and screening of the data to be processed and improving the accuracy of the data in the model dataset.

[0099] It should be noted that, if the selection result of the data to be processed is to retain the data to be processed, the comprehensive score of the data to be processed can be determined based on the second detection result and the text information of the data to be processed, and then the importance of the data to be processed in the training data can be determined based on the comprehensive score.

[0100] When determining the comprehensive score, we first calculate the average score of each model based on the scores of the sub-detection results corresponding to each model in the second detection result, and then use the average scores of multiple models to calculate the overall average score to obtain the overall average score of the data to be processed. Furthermore, we use an independent large language model to determine the actual executability and safety of the planned movement actions or movement plans contained in the text content of the data to be processed, and obtain the execution score and safety score. Finally, we weight and sum the overall average score, execution score and safety score to obtain the comprehensive score of the data to be processed.

[0101] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0102] Example 2

[0103] This application also provides a data filtering device. It should be noted that the data filtering device of this application can be used to execute the data filtering method provided in the above embodiments. The data filtering device provided in this application is described below.

[0104] According to an embodiment of this application, an apparatus for implementing the above-described data filtering method is also provided. Figure 3 This is a schematic diagram of the data filtering device provided in Embodiment 2 of this application, as shown below. Figure 3 As shown, the device includes:

[0105] The determining unit 31 is used to obtain the data to be processed from the database and determine the data type and application scenario of the data to be processed, wherein the data to be processed is single-turn dialogue data or multi-turn dialogue data.

[0106] The detection unit 32 is used to determine the target detection strategy for the data to be processed according to the application scenario, and to detect the data to be processed according to the target detection strategy to obtain the first detection result.

[0107] The acquisition unit 33 is used to obtain the detection items associated with the data to be processed from the data detection table according to the application scenario and data type when the first detection result indicates that there are no abnormalities in the data to be processed, thereby obtaining a set of target detection items.

[0108] The processing unit 34 is used to input the target detection item set and the data to be processed into a preset large language model set to obtain the second detection result, and to determine the filtering operation to be performed on the data to be processed based on the second detection result.

[0109] The data filtering device provided in this application embodiment obtains data to be processed from the database by the determining unit 31, and determines the data type and application scenario of the data to be processed, wherein the data to be processed is single-turn dialogue data or multi-turn dialogue data; the detection unit 32 determines the target detection strategy of the data to be processed according to the application scenario, and detects the data to be processed according to the target detection strategy to obtain a first detection result; the acquisition unit 33, when the first detection result indicates that the data to be processed is not abnormal, obtains the detection items associated with the data to be processed from the data detection table according to the application scenario and data type to obtain a target detection item set; the processing unit 34 inputs the target detection item set and the data to be processed into a preset large language model set to obtain a second detection result, and determines the filtering operation to be performed on the data to be processed according to the second detection result. By determining the target detection strategy based on the data type and application scenario of the data to be detected, the data to be detected is detected once. Then, the detection items are determined based on the application scenario and data type, and the data to be detected is detected a second time using the detection items. This determines whether the data to be detected needs to be filtered, thereby improving the accuracy of data quality detection and filtering. This solves the technical problem in related technologies where the accuracy of data filtering operations is low because they rely solely on text quality indicators for language-level detection and filtering.

[0110] Optionally, in the data filtering device provided in this application embodiment, the detection unit 32 includes: a first acquisition module, used to acquire detection strategies associated with the application scenario from a strategy library to obtain a first detection strategy; a second acquisition module, used to acquire the data content of the data to be processed and determine the feature information of the data to be processed based on the data content; a first determination module, used to determine the feature type to which the feature information belongs and select a second detection strategy from the first detection strategy based on the feature type, wherein the second detection strategy contains detection content corresponding to the feature type; and determine the second detection strategy as the target detection strategy.

[0111] Optionally, in the data filtering device provided in this application embodiment, the target detection strategy includes M sub-detection strategies, and the detection unit 32 includes: a detection module, used to detect the data to be processed according to each sub-detection strategy to obtain M first sub-detection results, where M is a positive integer; a first judgment module, used to judge whether there is a target sub-detection result with an anomaly level of the first level among the M first sub-detection results; a second determination module, used to determine that the first detection result is abnormal if there is a target sub-detection result among the M first sub-detection results; and a third determination module, used to determine that the first detection result is not abnormal if there is no target sub-detection result among the M first sub-detection results.

[0112] Optionally, in the data filtering device provided in the embodiments of this application, the device further includes: a deletion unit, used to determine the filtering operation to be performed on the data to be processed as deleting the data to be processed from the database when the first detection result indicates that the data to be processed is abnormal.

[0113] Optionally, in the data filtering device provided in this application embodiment, the acquisition unit 33 includes: a third acquisition module, used to acquire general detection items from the data detection table and acquire scene detection items associated with the application scenario; a fourth acquisition module, used to acquire a set of condition information associated with the data detection table and determine the target condition information that the data to be processed meets; a fifth acquisition module, used to acquire condition detection items corresponding to the target condition information from the data detection table and acquire round detection items associated with the data type; and a construction module, used to construct a target detection item set based on the general detection items, scene detection items, condition detection items, and round detection items.

[0114] Optionally, in the data filtering device provided in this application embodiment, the preset large language model set includes N preset large language models, and the processing unit 34 includes: a processing module, used to input the target detection item set and the data to be processed into the preset large language model respectively for any preset large language model to obtain a second sub-detection result, wherein the second sub-detection result includes a detection score set, the detection score set including the detection scores of the preset large language model for detecting each target detection item in the target detection item set; a second judgment module, used to determine whether there are scores less than or equal to a target threshold in the detection score set of the second sub-detection result; a fourth determination module, used to determine that the second sub-detection result is abnormal if there are scores less than or equal to the target threshold in the detection score set; a fifth determination module, used to determine that the second sub-detection result is not abnormal if there are no scores less than or equal to the target threshold in the detection score set; and a combination module, used to combine the second sub-detection results of each preset large language model to obtain a second detection result.

[0115] Optionally, in the data filtering device provided in this application embodiment, the processing unit 34 includes: a third judgment module, used to judge whether there is a second sub-detection result with an abnormal characterization in the second detection result; a sixth determination module, used to determine the filtering operation to be performed on the data to be processed as deleting the data to be processed from the database when there is a second sub-detection result with an abnormal characterization in the second detection result; and a seventh determination module, used to determine the filtering operation to be performed on the data to be processed as retaining the data to be processed in the database when there is no second sub-detection result with an abnormal characterization in the second detection result.

[0116] It should be noted that the aforementioned determining unit 31, detecting unit 32, acquiring unit 33, and processing unit 34 correspond to steps S201 to S204 in Embodiment 1. The instances and application scenarios implemented by each of these units and their corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1. It should also be noted that the aforementioned modules or units can be hardware or software components stored in a memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). These modules can also run as part of a device in the computer terminal 10 provided in Embodiment 1.

[0117] Example 3

[0118] Embodiments of this application may provide an electronic device. Figure 4 This is a structural block diagram of an electronic device according to an embodiment of this application. Figure 4 As shown, the electronic device may include: one or more ( Figure 4 (Only one is shown) processor 1002, memory 1004, memory controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0119] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and apparatus in the embodiments of this application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the above-described methods. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0120] Those skilled in the art will understand that Figure 4 The structure shown is for illustrative purposes only. Electronic devices can also be smartphones, tablets, handheld computers, mobile internet devices (MIDs), PADs, and other terminal devices. Figure 4 This does not limit the structure of the aforementioned electronic device. For example, electronic devices may also include components that are more... Figure 4 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 4 The different configurations shown.

[0121] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0122] Example 4

[0123] Embodiments of this application also provide a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the data filtering method provided in Embodiment 1.

[0124] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0125] Embodiments of this application also provide a computer program product, which, when executed on a data processing device, is adapted to perform the steps of a data filtering method.

[0126] Embodiments of this application also provide a computer-readable storage medium, which includes a stored executable program, wherein the executable program controls the device where the computer-readable storage medium is located to perform the above-described data filtering method when it runs.

[0127] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0128] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0129] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0130] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0131] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0132] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0133] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A data filtering method, characterized in that, include: Retrieve data to be processed from the database and determine the data type and application scenario of the data to be processed, wherein the data to be processed is single-turn dialogue data or multi-turn dialogue data; Based on the application scenario, a target detection strategy for the data to be processed is determined, and the data to be processed is detected according to the target detection strategy to obtain a first detection result; If the first detection result indicates that the data to be processed is not abnormal, the detection items associated with the data to be processed are obtained from the data detection table according to the application scenario and the data type to obtain the target detection item set; The target detection item set and the data to be processed are input into a preset large language model set to obtain a second detection result, and the filtering operation to be performed on the data to be processed is determined based on the second detection result.

2. The method according to claim 1, characterized in that, The target detection strategy for the data to be processed, determined according to the application scenario, includes: The first detection strategy is obtained by retrieving the detection strategy associated with the application scenario from the strategy library. Obtain the data content of the data to be processed, and determine the feature information of the data to be processed based on the data content; Determine the feature type to which the feature information belongs, and select a second detection strategy from the first detection strategy according to the feature type, wherein the second detection strategy includes the detection content corresponding to the feature type; The second detection strategy is determined as the target detection strategy.

3. The method according to claim 1, characterized in that, The target detection strategy includes M sub-detection strategies. The data to be processed is detected according to the target detection strategies to obtain a first detection result, including: The data to be processed is detected according to each sub-detection strategy to obtain M first sub-detection results, where M is a positive integer; Determine whether there is a target sub-detection result with an anomaly level of first among the M first sub-detection results; If the target sub-detection result exists among the M first sub-detection results, the first detection result is determined to be abnormal. If the target sub-detection result is not found among the M first sub-detection results, the first detection result is determined to be free of anomalies.

4. The method according to claim 1, characterized in that, The method further includes: If the first detection result indicates that the data to be processed is abnormal, the filtering operation to be performed on the data to be processed will be determined as deleting the data to be processed from the database.

5. The method according to claim 1, characterized in that, Based on the application scenario and the data type, the detection items associated with the data to be processed are obtained from the data detection table, resulting in a target detection item set including: Obtain general detection items from the data detection table, and obtain scene detection items associated with the application scenario; Obtain the set of condition information associated with the data detection table, and determine the target condition information that the data to be processed meets; Obtain the condition detection items corresponding to the target condition information from the data detection table, and obtain the round detection items associated with the data type; The target detection item set is constructed based on the general detection items, the scene detection items, the condition detection items, and the round detection items.

6. The method according to claim 1, characterized in that, The preset large language model set includes N preset large language models. Inputting the target detection item set and the data to be processed into the preset large language model set yields the following second detection results: For any preset large language model, the target detection item set and the data to be processed are respectively input into the preset large language model to obtain a second sub-detection result. The second sub-detection result includes a detection score set, which includes the detection scores of the preset large language model for each target detection item in the target detection item set. Determine whether there exists a score less than or equal to the target threshold in the set of detected scores in the second sub-detection result; If there is a score in the set of detected scores that is less than or equal to the target threshold, the second sub-detection result is determined to be abnormal. If there is no score less than or equal to the target threshold in the set of detected scores, the second sub-detection result is determined to be without anomaly; The second detection result is obtained by combining the second sub-detection results of each preset large language model.

7. The method according to claim 6, characterized in that, The filtering operations to be performed on the data to be processed based on the second detection result include: Determine whether there is a second sub-detection result that represents an anomaly in the second detection result; If a second sub-detection result with an abnormal character is found in the second detection result, the filtering operation to be performed on the data to be processed will be determined as deleting the data to be processed from the database; If no second sub-detection result indicating an anomaly is found in the second detection result, the filtering operation performed on the data to be processed will be determined to retain the data to be processed in the database.

8. A data filtering device, characterized in that, include: A determining unit is used to obtain data to be processed from the database and determine the data type and application scenario of the data to be processed, wherein the data to be processed is single-turn dialogue data or multi-turn dialogue data; The detection unit is used to determine the target detection strategy for the data to be processed according to the application scenario, and to detect the data to be processed according to the target detection strategy to obtain a first detection result; The acquisition unit is used to obtain the detection items associated with the data to be processed from the data detection table according to the application scenario and the data type when the first detection result indicates that the data to be processed is not abnormal, thereby obtaining a target detection item set. The processing unit is used to input the target detection item set and the data to be processed into a preset large language model set to obtain a second detection result, and to determine the filtering operation to be performed on the data to be processed based on the second detection result.

9. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the steps of the data filtering method according to any one of claims 1 to 7.

10. An electronic device, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, performs the data filtering method according to any one of claims 1 to 7.

Images