Data set refining method, system and equipment and storage medium
By employing semantic clustering and question-answer pair elimination strategies, the problem of uncontrollable dataset quality is solved, enabling efficient dataset filtering and refinement, improving the accuracy and stability of the dataset, and making it suitable for model fine-tuning and application deployment.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-28
- Publication Date
- 2026-03-13
AI Technical Summary
Existing dataset construction methods lack a comprehensive assessment of the accuracy, completeness, and stability of question-answer pairs, resulting in uncontrollable dataset quality.
By using semantic clustering and elimination strategies for question-answer pairs, question-answer pairs of the same type are first grouped into the same cluster. Then, the corresponding elimination strategies are used for filtering. Finally, similar datasets are randomly assigned to different alliance queues for comparison and iterative output, thereby improving the efficiency and accuracy of the filtering process.
It improves the efficiency of dataset refinement and engineering accuracy, ensures the quality control of the dataset, and is suitable for model fine-tuning and application deployment.
Smart Images
Figure CN121658602A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, system, device and storage medium for refining datasets. Background Technology
[0002] With the development of large-scale model technology, various industries are actively fine-tuning and implementing models. The quality of the dataset directly determines the effectiveness of model fine-tuning. Existing methods mostly rely on semantic clustering, inverted indexes, and other techniques, lacking a comprehensive evaluation of the accuracy, completeness, and stability of question-answer pairs, resulting in uncontrollable dataset quality. For example, the semantic clustering deduplication method proposed by Volcano Engine still suffers from the aforementioned problems. Summary of the Invention
[0003] This application aims to at least address the technical problems existing in the prior art and provide a data set refinement method, system, device, and storage medium.
[0004] In a first aspect, the present invention provides a dataset refinement method, the method comprising: Acquire the question and answer data to be processed, which includes the question information raised by the questioner and the answer information replied by the respondent; Parse the question-and-answer data, extract question-and-answer pairs from the data, and generate an initial question-and-answer pair pool; Capture the semantic information of each question-answer pair in the initial question-answer pair pool, and divide the question-answer pairs in the initial question-answer pair pool into several clusters based on the semantic information of each question-answer pair; For each question-answer pair in a cluster, the corresponding elimination strategy is used to process the question-answer pairs in the cluster, resulting in a filtered set of question-answer pairs. The data in the filtered question-and-answer set is confirmed and corrected to obtain a refined dataset.
[0005] By adopting the above technical solution, this application first divides question-answer pairs of the same type into the same cluster. For question-answer pairs of the same type, a matching elimination strategy is used to filter the question-answer pairs in the cluster, thereby improving the efficiency of filtering question-answer pairs in the dataset that do not meet the requirements.
[0006] Optionally, the step of capturing the semantic information of each question-answer pair in the initial question-answer pair pool, and dividing the question-answer pairs in the initial question-answer pair pool into several clusters based on the semantic information of each question-answer pair, includes: Semantic features of each question-answer pair in the initial question-answer pair pool are extracted to obtain a joint vector. The semantic information of the question-answer pair is represented by the joint vector. Normalize the joint vector of each question-answer pair to obtain the normalized joint vector; HDBSCAN density clustering is used to map the normalized joint vector to the corresponding cluster, so that the question-answer pairs with the smallest overall semantic distance within the cluster are placed in the same cluster.
[0007] By adopting the above technical solution, the question-answer pairs in the initial question-answer pair pool are clustered, and question-answer pairs of the same type are placed in the same cluster. The same elimination strategy is used to filter question-answer pairs of the same type, thereby improving the efficiency of dataset refinement.
[0008] Optionally, question-answer pairs within the same cluster are defined as the same type of question-answer pairs. The step of processing question-answer pairs within each cluster using a corresponding elimination strategy to obtain a filtered set of question-answer pairs includes: The same type of question-answer pairs are randomly placed into two alliances, and adversarial comprehensive confidence is calculated to obtain the comprehensive confidence. The comparison results are obtained by comparing the overall confidence of the question-answer pairs in the two groups of alliances. Based on the comparison results, it is determined whether the question-answer pairs should be eliminated. The question-answer pairs that need to be eliminated are deleted from the cluster, and the filtered question-answer pair set is obtained.
[0009] By adopting the above technical solution, similar datasets are randomly assigned to different consortium queues for comparison, and iterative output is performed, which can further improve the efficiency of dataset selection and the accuracy of engineering.
[0010] Optionally, the formula for calculating the overall confidence level is: ; in, Indicates the overall confidence level. This represents the stability score of the question-and-answer pair. The score represents the completeness of information in the question-and-answer pair. The score indicates the accuracy of the question-and-answer pair. Indicates the stability score weight. The weights of the information integrity score are represented. The score indicates the accuracy of the correct answer.
[0011] By adopting the above technical solution, the specific calculation method of the comprehensive confidence level has been clarified.
[0012] Optionally, the formula for calculating the stability score of the question-answer pair is: ; in, Represents Shannon entropy; calculates the percentage of identical words appearing in the answers of each question-and-answer pair. This represents the reciprocal of the percentage value; ; This represents the keyword index of the question-answer pairs in the cluster. This represents the total number of keywords in the question-answer pairs within the cluster. Keywords Word frequency.
[0013] By adopting the above technical solution, the calculation method of the stability score of question-answer pairs was clarified.
[0014] Optionally, the formula for calculating the information completeness score of a question-and-answer pair is: ; in, R represents the proportion of entity-relationships extracted from question-answer pairs that can be found in the knowledge graph, and R represents the proportion of entity-relationships extracted from question-answer pairs that should appear in the knowledge graph.
[0015] By adopting the above technical solution, the calculation method for the information integrity score of question-answer pairs has been clarified.
[0016] Optionally, the accuracy score of the question-answer pair can be obtained by calculating the overlap between the answer given by the question-answer pair and the standard answer at the longest common subsequence level using the ROUGE-L F1 algorithm.
[0017] By adopting the above technical solution, the calculation method for the accuracy score of question-answer pairs has been clarified.
[0018] Secondly, the present invention provides a dataset refinement system, the system comprising: The acquisition module is used to acquire the question and answer data to be processed. The question and answer data includes the question information raised by the questioner and the answer information replied by the answerer. The parsing module is used to parse question-and-answer data, extract question-and-answer pairs from the data, and generate an initial question-and-answer pair pool. The clustering module is used to capture the semantic information of each question-answer pair in the initial question-answer pair pool and divide the question-answer pairs in the initial question-answer pair pool into several clusters based on the semantic information of each question-answer pair. The filtering module is used to process the question-answer pairs in each cluster using the corresponding elimination strategy, and obtain the filtered question-answer pair set. The output module is used to confirm and correct the data in the filtered question-and-answer pair set to obtain a refined dataset.
[0019] Thirdly, the present invention provides an electronic device, the electronic device comprising: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the dataset refining method described above.
[0020] Fourthly, the present invention also provides a computer-readable storage medium storing at least one computer program, which is executed by a processor in an electronic device to implement the dataset refining method described above.
[0021] In summary, this application includes the following beneficial technical effects: This application first divides question-answer pairs of the same type into the same cluster. For question-answer pairs of the same type, a matching elimination strategy is used to filter the question-answer pairs in the cluster, thereby improving the efficiency of filtering question-answer pairs in the dataset that do not meet the requirements. Furthermore, this invention proposes to randomly assign similar datasets into different alliance queues for comparison, while simultaneously performing iterative output, which can further improve the efficiency of dataset selection and the accuracy of engineering. Attached Figure Description
[0022] Figure 1 This is a flowchart illustrating a dataset refinement method provided in an embodiment of the present invention. Figure 2 A flowchart of an embodiment of the present invention is provided for randomly placing the same type of question-answer pairs into two alliances and performing adversarial comprehensive confidence calculation; Figure 3 This is a schematic diagram of a dataset refining process provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of an electronic device that implements the dataset refining method according to an embodiment of the present invention.
[0023] Reference numerals: 10, processor; 11, memory; 12, communication bus; 13, communication interface.
[0024] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0025] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.
[0026] In the description of this invention, it should be understood that the terms "longitudinal", "lateral", "up", "down", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.
[0027] In the description of this invention, unless otherwise specified and limited, it should be noted that the terms "installation", "connection" and "linking" should be interpreted broadly. For example, they can refer to mechanical or electrical connections, or internal connections between two components. They can be direct connections or indirect connections through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms according to the specific circumstances.
[0028] Reference Figure 1 The diagram shown is a flowchart illustrating a dataset refinement method according to an embodiment of the present invention. In this embodiment, the dataset refinement method includes: S1. Obtain the question and answer data to be processed. The question and answer data includes the question information raised by the questioner and the answer information replied by the respondent.
[0029] Question and answer data is obtained by extracting a carrier, which can be a file, voice, video, business database or data warehouse, etc. In this embodiment, the extracted carrier includes, but is not limited to, common text electronic carriers such as txt, csv, excel, pdf, word, and Markdown.
[0030] S2. Parse the question-and-answer data, extract question-and-answer pairs from the data, and generate an initial question-and-answer pair pool.
[0031] When extracting the carrier, it provides segmented text processing for files such as txt / pdf / word / Markdown, for example, it segments paragraphs according to " / n" to form chapter and paragraph information, provides a rule engine (first sentence or last sentence) to form key information points, and combines AI language models (such as deepseekR1) to convert the carrier into "question-answer" pairs (referred to as question-answer pairs).
[0032] For files such as CSV and Excel, data can be imported into the data warehouse. Semantic information can be extracted using SQL combined with regular expressions to obtain question-answer pairs. The business database and data warehouse also extract semantic information using SQL combined with regular expressions to obtain question-answer pairs.
[0033] For audio and video messages (such as voice messages from human agents), speech recognition technology is used to convert the audio in the audio and video messages into text output and store it in a CSV file. By parsing the converted text, the system identifies the questioner and the customer service representative who is responding, thus forming a question-and-answer pair.
[0034] The above methods are used to form an initial question-and-answer pair pool.
[0035] S3. Capture the semantic information of each question-answer pair in the initial question-answer pair pool, and divide the question-answer pairs in the initial question-answer pair pool into several clusters based on the semantic information of each question-answer pair.
[0036] Questions and answers are categorized through similarity calculations or manual judgment and automatically combined with topics provided by the device. The semantics of "question" and "answer" are captured to avoid misclassification of "same questions but different answers" or "same answers but different questions". In addition, the elimination calculation is carried out in subsequent steps by using clusters as units, which can compress the original huge initial pool into computable clusters, greatly reducing the data volume of subsequent calculations.
[0037] Specifically, the semantic information of each question-answer pair in the initial question-answer pair pool is captured, and the question-answer pairs in the initial question-answer pair pool are divided into several clusters based on the semantic information of each question-answer pair, including: S31. Extract the semantic features of each question-answer pair in the initial question-answer pair pool to obtain a joint vector. The semantic information of the question-answer pair is represented by the joint vector.
[0038] Let the initial question-answer pool be D0. Generate a joint vector for each question-answer pair in the initial question-answer pool D0: ; in, This indicates the processing of problem side vectors. This indicates the processing of the answer's lateral vector. To indicate a problem, Indicate the answer, and Both adopt the SimCSE vector representation framework and use a 768-dimensional model for computation; ⊕ means concatenation, which means concatenating the terms on the question side and the answer side into a 1536-dimensional term.
[0039] S32. Normalize the joint vector of each question-answer pair to obtain the normalized joint vector.
[0040] By performing an L2 scaling operation on the joint vector v, forcibly reducing the vector length to 1, the range of cosine similarity calculations will be fixed in [0, 2] (cosine distance = 1). cosine similarity, similarity ∈ [ [1, 1]⇒ distance ∈ [0, 2]), to facilitate subsequent processing of the joint vector.
[0041] S33. Use HDBSCAN density clustering to map the normalized joint vector to the corresponding cluster, so that the question-answer pairs with the smallest overall semantic distance within the cluster are placed in the same cluster.
[0042] HDBSCAN is an advanced hierarchical density clustering algorithm that can automatically discover clusters of arbitrary shapes in data and effectively identify noise points. It only requires setting a single, intuitive parameter—the minimum cluster size (min_cluster_size)—to obtain meaningful clustering results, eliminating the need for repeated adjustments to the epsilon parameter. We focus on its core parameters: min_samples is the minimum radius required for "core points," and points that do not reach min_samples are temporarily marked as potential noise points. min_cluster_size is the minimum number of points in an "effective cluster" (i.e., the minimum cluster size), avoiding the generation of too many meaningless small clusters or clusters that cannot be identified.
[0043] S4. For each question-answer pair in a cluster, use the corresponding elimination strategy to process the question-answer pairs in the cluster, and obtain the filtered question-answer pair set.
[0044] Defining question-answer pairs within the same cluster as belonging to the same category, specifically, the steps for processing question-answer pairs within each cluster using the corresponding elimination strategy to obtain the filtered set of question-answer pairs include: S41. Randomly place the same type of question-answer pairs into two alliances, perform adversarial comprehensive confidence calculation, and obtain the comprehensive confidence score.
[0045] Reference Figure 2 For question-answer pairs of the same type, randomly place them into two consortia and perform adversarial comprehensive confidence calculation. Based on the cluster {Cj} obtained in step 3, for each cluster question-answer set Dt, randomly place it into two consortia L1 and L2, and calculate the comprehensive confidence G.
[0046] Reference Figure 3 For each question-answer pair, three indicators (stability score, information integrity score, and accuracy score) are coupled and quantified to obtain the comprehensive confidence score.
[0047] The formula for calculating the overall confidence level is: ; in, Indicates the overall confidence level. This represents the stability score of the question-and-answer pair. The score represents the completeness of information in the question-and-answer pair. The score indicates the accuracy of the question-and-answer pair. Indicates the stability score weight. The weights of the information integrity score are represented. The score indicates the accuracy of the correct answer.
[0048] The formula for calculating the stability score of each question-answer pair is as follows: ; in, Represents Shannon entropy; calculates the percentage of identical words appearing in the answers of each question-and-answer pair. This represents the reciprocal of the percentage value; For grouped alliances L1 and L2, similar grouped question-and-answer sets Statistical frequency is used to calculate Shannon entropy H using the following formula: ; This represents the keyword index of the question-answer pairs in the cluster. This represents the total number of keywords in the question-answer pairs within the cluster. Keywords Word frequency.
[0049] Specifically, the frequency of words for the same type of problem is statistically analyzed: for example, the word (traffic) appears m times, the word (car) appears n times, and so on, the word (injury / fatality) appears l times, then the total number of occurrences sum = m + n + ... + l, p(traffic) = m / sum, p(car) = n / sum, etc. H is calculated based on this data.
[0050] Stability score of question-answer pairs For the grouped question-answer set Dt_i, count the results for each question-answer pair.<q,a> The K-value is the reciprocal of the percentage of identical words appearing in the answers. For example: If there are 10 answers in consortium L1, and the answer frequencies of p1, p2, p4, and p5 in consortium L1 are the same, then the proportion is 4 / 10 = 0.4, and the reciprocal of the proportion, K, is 2.5. If the number of identical answers in p6 is 1, then K = 10. The value range of S is [0, 1], that is, [most unstable, most stable].
[0051] When calculating the information completeness score C of a question-and-answer pair, extract the question-and-answer pairs.<q,a> Treating the "entity-relationship" pair as a miniature knowledge graph extraction test, the formula for calculating the information completeness score of the question-answer pair is: ; in, R represents the proportion of entity-relationships extracted from question-answer pairs that can be found in the knowledge graph, and R represents the proportion of entity-relationships extracted from question-answer pairs that should appear in the knowledge graph.
[0052] Use information extraction models (such as UIE, DeepKE, PaddleNLP-UIE) to process question-answer pairs.<q,a> Extract the triple T = {(e1, r, e2)}. Further, in the Gold knowledge graph, use the extracted entities as anchor points to pull a 1-hop subgraph, thus obtaining the set of triples that the sentence should contain. After entity linking and alignment, calculate P=|T∩ | / |T|,R=|T∩ | / | The information integrity score C is calculated according to the above formula.
[0053] The ROUGE-L F1 algorithm is used to calculate the overlap between the answers given by the question-answer pair and the standard answer at the longest common subsequence level. This algorithm is specifically suitable for evaluating the fidelity of natural language sentences and obtaining an accuracy score for the question-answer pair. The specific steps include: S411. Denote the candidate answer as a (the candidate answer is the model output or the answer to be evaluated) and prepare a reference answer a*. There can be multiple reference answers a* or just one reference answer a*. The reference answer a* is set according to the Gold standard knowledge graph and the highest score is taken.
[0054] S412, ROUGE-L similarity calculation: Let LCS(a, a*) be the longest common subsequence of a and a* (preserving word order).
[0055] Recall rate =|LCS| / |a*; Accuracy =|LCS| / |a|.
[0056] S413. If the gold knowledge graph provides n references, then A = max_{i=1,...,n} (a, ) S414. The A value is obtained based on the overlap between the answer given by the question-and-answer pair and the standard answer at the longest common subsequence level. A∈[0,1], where 1 indicates that it is completely consistent with the reference answer (LCS coverage 100%).
[0057] S42. Compare the overall confidence of the question-answer pairs in the two groups to obtain the comparison results. Determine whether the question-answer pairs should be eliminated based on the comparison results. Remove the question-answer pairs that need to be eliminated from the cluster to obtain the filtered question-answer pair set.
[0058] A strategy for eliminating question-answer pairs is specified, and iterative refinement steps are performed to ultimately form a high-quality dataset. The elimination strategy is set as follows: if the Top-K question-answer pairs are swapped, and a question-answer pair's G value in the opposing alliance is lower than θ = 20% of its own alliance's G value (this value θ can be set in the device), then it is eliminated. This process is repeated in step 4 until the latest set of question-answer pairs is reached. ≤30% of the initial pool size Dt or the number of iterations: =10 times, output the filtered question-answer pair set Z.
[0059] S5. Confirm and correct the data in the filtered question-and-answer pair set to obtain a refined dataset.
[0060] Through the interface provided by the device, manual intervention and screening are conducted to create a knowledge base that can be integrated with models. Based on the filtered question-answer pair set, the device interface allows data engineers to query, screen, confirm, and correct the dataset by topic, ensuring that the dataset is more complete and refined under human intervention.
[0061] After obtaining the refined dataset, dataset refinement methods also include: The data themes are determined based on the content of the refined dataset, and the refined dataset is bound to the data themes. The data themes include at least one of the following: government affairs themes, transportation themes, and accident themes. The data themes can be entered by business experts through the device as a theme domain tree, similar to government affairs themes, transportation themes, and accident themes. The theme domain tree vector is stored in the device database, and the dataset only needs to query the database according to the statement vector to bind the relevant themes.
[0062] Based on the same inventive concept, an embodiment of the present invention provides a dataset refining system.
[0063] The dataset refining system of this invention can be installed in an electronic device. Depending on the functions implemented, the dataset refining system includes: The acquisition module is used to acquire the question and answer data to be processed. The question and answer data includes the question information raised by the questioner and the answer information replied by the answerer. The parsing module is used to parse question-and-answer data, extract question-and-answer pairs from the data, and generate an initial question-and-answer pair pool. The clustering module is used to capture the semantic information of each question-answer pair in the initial question-answer pair pool and divide the question-answer pairs in the initial question-answer pair pool into several clusters based on the semantic information of each question-answer pair. The filtering module is used to process the question-answer pairs in each cluster using the corresponding elimination strategy, and obtain the filtered question-answer pair set. The output module is used to confirm and correct the data in the filtered question-and-answer pair set to obtain a refined dataset.
[0064] The module described in this invention can also be called a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and are stored in the memory of the electronic device.
[0065] The various variations and specific examples of the dataset refining method provided in the above embodiments are also applicable to the dataset refining system of this embodiment. Through the foregoing detailed description of the dataset refining method, those skilled in the art can clearly understand the implementation method of the dataset refining system in this embodiment. For the sake of brevity, they will not be described in detail here.
[0066] This application also discloses an electronic device, such as Figure 4 The diagram shown is a schematic representation of an electronic device for a dataset refining method according to an embodiment of the present invention. The electronic device may include at least one processor 10, a memory 11 communicatively connected to the at least one processor, a communication bus 12, and a communication interface 13. It may also include a computer program, such as a dataset refining method program, stored in the memory 11 and executable on the processor 10.
[0067] In some embodiments, the processor 10 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It performs various functions of the electronic device and processes data by running or executing programs or modules stored in the memory 11 (e.g., methods for refining datasets), and by calling data stored in the memory 11.
[0068] The memory 11 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of an electronic device, such as a portable hard drive. In other embodiments, the memory 11 can be an external storage device of the electronic device, such as a plug-in portable hard drive, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc. Furthermore, the memory 11 can include both internal and external storage units of the electronic device. The memory 11 can be used not only to store application software and various types of data installed on the electronic device, such as code for dataset refinement methods, but also to temporarily store data that has been output or will be output.
[0069] The communication bus 12 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable communication between the memory 11 and at least one processor 10, etc.
[0070] Communication interface 13 is used for communication between the aforementioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, Bluetooth interface, etc.), typically used to establish communication connections between the electronic device and other electronic devices. The user interface may be a display, an input unit (such as a keyboard), or optionally, a standard wired or wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device and to display a visual user interface.
[0071] Figure 4 Only electronic devices with components are shown; it will be understood by those skilled in the art that... Figure 4The structure shown does not constitute a limitation on the electronic device and may include fewer or more components than shown, or combine certain components, or have different component arrangements.
[0072] For example, although not shown, the electronic device may also include a power supply (such as a battery) to power various components. Preferably, the power supply can be logically connected to at least one processor 10 via a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.
[0073] It should be understood that the embodiments are for illustrative purposes only and are not limited to this structure in the scope of the patent application.
[0074] Furthermore, if the modules / units integrated into the electronic device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile.
[0075] This application provides a computer-readable storage medium, including, for example, any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM). The computer-readable storage medium stores a computer program that can be loaded by a processor and execute the dataset refining method of the above embodiments.
[0076] In the description of this specification, the references to terms such as "an embodiment," "some embodiments," "example," "specific example," "a implementation," "a preferred implementation," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.
[0077] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.
Claims
1. A dataset refinement method, characterized in that, The method includes: Acquire the question and answer data to be processed, which includes the question information raised by the questioner and the answer information replied by the respondent; Parse the question-and-answer data, extract question-and-answer pairs from the data, and generate an initial question-and-answer pair pool; Capture the semantic information of each question-answer pair in the initial question-answer pair pool, and divide the question-answer pairs in the initial question-answer pair pool into several clusters based on the semantic information of each question-answer pair; For each question-answer pair in a cluster, the corresponding elimination strategy is used to process the question-answer pairs in the cluster, resulting in a filtered set of question-answer pairs. The data in the filtered question-and-answer set is confirmed and corrected to obtain a refined dataset.
2. The dataset refinement method as described in claim 1, characterized in that, The process involves capturing the semantic information of each question-answer pair in the initial question-answer pair pool, and dividing the question-answer pairs in the initial pool into several clusters based on the semantic information of each pair, including: Semantic features of each question-answer pair in the initial question-answer pair pool are extracted to obtain a joint vector. The semantic information of the question-answer pair is represented by the joint vector. Normalize the joint vector of each question-answer pair to obtain the normalized joint vector; HDBSCAN density clustering is used to map the normalized joint vector to the corresponding cluster, so that the question-answer pairs with the smallest overall semantic distance within the cluster are placed in the same cluster.
3. The dataset refinement method as described in claim 1, characterized in that, Question-answer pairs within the same cluster are defined as belonging to the same category. The process involves using a corresponding elimination strategy to process question-answer pairs within each cluster, resulting in a filtered set of question-answer pairs, including: The same type of question-answer pairs are randomly placed into two alliances, and adversarial comprehensive confidence is calculated to obtain the comprehensive confidence. The comparison results are obtained by comparing the overall confidence of the question-answer pairs in the two groups of alliances. Based on the comparison results, it is determined whether the question-answer pairs should be eliminated. The question-answer pairs that need to be eliminated are deleted from the cluster, and the filtered question-answer pair set is obtained.
4. The dataset refining method as described in claim 3, characterized in that, The formula for calculating the overall confidence level is: ； in, Indicates the overall confidence level. This represents the stability score of the question-and-answer pair. The score represents the completeness of information in the question-and-answer pair. The score indicates the accuracy of the question-and-answer pair. Indicates the stability score weight. The weights of the information integrity score are represented. The score indicates the accuracy of the question and answer.
5. The dataset refinement method as described in claim 4, characterized in that, The formula for calculating the stability score of question-answer pairs is: ； in, Represents Shannon entropy; calculates the percentage of identical words appearing in the answers of each question-and-answer pair. This represents the reciprocal of the percentage value; ； This represents the keyword index of the question-answer pairs in the cluster. This represents the total number of keywords in the question-answer pairs within the cluster. Keywords Word frequency.
6. The dataset refining method as described in claim 4, characterized in that, The formula for calculating the information completeness score of a question-and-answer pair is: ； in, R represents the proportion of entity-relationships extracted from question-answer pairs that can be found in the knowledge graph, and R represents the proportion of entity-relationships extracted from question-answer pairs that should appear in the knowledge graph.
7. The dataset refining method as described in claim 4, characterized in that, The accuracy score of the question-answer pair is obtained by calculating the overlap between the answer given by the question-answer pair and the standard answer at the longest common subsequence level using the ROUGE-L F1 algorithm.
8. A dataset refinement system for implementing the dataset refinement method according to any one of claims 1 to 7, characterized in that, include: The acquisition module is used to acquire the question and answer data to be processed. The question and answer data includes the question information raised by the questioner and the answer information replied by the answerer. The parsing module is used to parse question-and-answer data, extract question-and-answer pairs from the data, and generate an initial question-and-answer pair pool. The clustering module is used to capture the semantic information of each question-answer pair in the initial question-answer pair pool and divide the question-answer pairs in the initial question-answer pair pool into several clusters based on the semantic information of each question-answer pair. The filtering module is used to process the question-answer pairs in each cluster using the corresponding elimination strategy, and obtain the filtered question-answer pair set. The output module is used to confirm and correct the data in the filtered question-and-answer pair set to obtain a refined dataset.
9. An electronic device, characterized in that, The electronic device includes: At least one processor (10); and, A memory (11) communicatively connected to the at least one processor (10); The memory (11) stores a computer program that can be executed by the at least one processor (10) to enable the at least one processor (10) to perform the dataset refining method as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program; when executed by a processor, the computer program implements the dataset refining method as described in any one of claims 1 to 7.