Data processing method and device, equipment and storage medium
By generating question-and-answer pairs from questions and responses obtained from a knowledge platform and constructing an evaluation dataset, the problem of inaccurate model evaluation in existing technologies is solved, and accurate evaluation of the model's handling of complex real-world problems is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-24
- Publication Date
- 2026-03-10
AI Technical Summary
Existing technologies are insufficient in evaluating the ability of large language models to handle complex real-world problems, especially in dealing with long-tail knowledge that is highly specialized, scattered, and relatively sparse in the model's pre-training data. Existing methods cannot accurately reflect the model's capabilities.
Knowledge content is acquired from the knowledge platform of the target knowledge domain, question and response information is extracted, question-answer pairs are generated, and an evaluation dataset is built based on this content to evaluate the model.
It improves the accuracy of model evaluation, reflecting the model's ability to handle complex real-world problems, especially its ability to process long-tail knowledge.
Smart Images
Figure CN121637073A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present disclosure relates to the technical field of computer, and particularly relates to a data processing method and device, equipment and storage medium. BACKGROUND
[0002] With the development of computer technology, large language models are increasingly widely used in natural language processing, such as question answering, reasoning and code optimization. With the increasing complexity of model architecture and training strategy, how to accurately and comprehensively evaluate the real ability of the model has become a key problem to be solved.
[0003] At present, the method for evaluating the performance of the model is still insufficient in reflecting the ability of the model to process complex problems in the real world. In particular, the ability of the model to process long-tail knowledge that is highly specialized, scattered and relatively sparse in the pre-training data of the model. SUMMARY
[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a data processing method, device, equipment and storage medium.
[0005] In a first aspect, the embodiments of the present disclosure provide a data processing method, which comprises: obtaining target knowledge content from at least one knowledge platform corresponding to a target knowledge field; extracting a plurality of knowledge data groups from the target knowledge content; wherein the knowledge data groups comprise question information under the target knowledge field, at least one reply information corresponding to the question information; generating question and answer pair content corresponding to the knowledge data groups based on the question information and the reply information in the knowledge data groups and a preset question type; wherein the question and answer pair content comprises question content and answer content having a corresponding relationship, the question content is generated based on the question information in the knowledge data groups and the preset question type, and the answer content is generated based on the reply content and the preset question type; constructing an evaluation data set based on the question and answer pair content; wherein the evaluation data set is used to evaluate a target model.
[0006] In an optional implementation, the knowledge data groups further comprise context information corresponding to the question information, and the generating of the question and answer pair content corresponding to the knowledge data groups based on the question information and the reply information in the knowledge data groups and the preset question type comprises: Based on the question information, response information, context information, and preset question types in the knowledge data set, question-answer pair content corresponding to the knowledge data set is generated; wherein, the question-answer pair content also includes reasoning explanation content corresponding to the question content, and the reasoning explanation content is used to explain the reasoning process from the question content to the corresponding answer content.
[0007] In an optional implementation, before generating the question-answer pair content corresponding to the knowledge data group based on the question information and response information in the knowledge data group and the preset question type, the method further includes: The question information in the multiple knowledge data groups is converted into semantic vectors respectively; Based on the semantic vectors corresponding to the question information in the multiple knowledge data groups, semantic deduplication is performed on the multiple knowledge data groups.
[0008] In one optional implementation, generating question-answer pairs corresponding to the knowledge data set based on the question and response information and preset question types in the knowledge data set includes: When the preset question types include multiple-choice questions, multiple-choice questions and multiple options are generated based on the question information and response information in the knowledge data group to constitute the question content; wherein, the multiple options include at least one correct option; Based on the response information in the knowledge data group, the answer content corresponding to the question content is generated, and a question-answer pair is constructed based on the answer content and the question content.
[0009] In one optional implementation, generating question-answer pairs corresponding to the knowledge data set based on the question and response information and preset question types in the knowledge data set includes: When the preset question types include short answer questions, question content is generated based on the question information in the knowledge data group, and answer content and test point content are generated based on the response information in the knowledge data group; wherein, the test point content is used to determine the accuracy of the answer to the question content; The question-and-answer pair is constructed based on the answer content, the question content, and the test point content.
[0010] In one optional implementation, obtaining the target knowledge content from at least one knowledge platform corresponding to the target knowledge domain includes: Obtain the page identifier of at least one knowledge page that meets the preset conditions from at least one knowledge platform corresponding to the target knowledge domain; Based on the page identifier, the target knowledge content is obtained from the knowledge page.
[0011] In one optional implementation, obtaining the target knowledge content from the knowledge page based on the page identifier includes: Based on the page identifier, screenshots are taken of the at least one knowledge page to obtain multiple knowledge content images; The extraction of multiple knowledge data groups from the target knowledge content includes: Multiple knowledge data sets are obtained by performing content recognition and semantic analysis on the multiple knowledge content images.
[0012] Secondly, this disclosure provides a data processing apparatus, the apparatus comprising: The first acquisition module is used to acquire target knowledge content from at least one knowledge platform corresponding to the target knowledge domain. The first extraction module is used to extract multiple knowledge data groups from the target knowledge content; wherein, the knowledge data group includes question information under the target knowledge domain and at least one response information corresponding to the question information; The first generation module is used to generate question-answer pairs corresponding to the knowledge data group based on the question information and response information in the knowledge data group and the preset question types; wherein, the question-answer pairs include question content and answer content with corresponding relationship, the question content is generated based on the question information in the knowledge data group and the preset question types, and the answer content is generated based on the response content and the preset question types; A construction module is used to build an evaluation dataset based on the question-and-answer pair content; wherein the evaluation dataset is used to evaluate the target model.
[0013] Thirdly, embodiments of this disclosure also provide an electronic device, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the data processing method provided in embodiments of this disclosure.
[0014] Fourthly, embodiments of this disclosure also provide a computer-readable storage medium storing a computer program for performing the data processing method provided in embodiments of this disclosure.
[0015] Fifthly, this disclosure provides a computer program product comprising a computer program / instruction that, when executed by a processor, implements the method described above.
[0016] The technical solution provided in this disclosure has at least the following advantages compared with the prior art: In the data processing method provided in this embodiment, firstly, target knowledge content is obtained from at least one knowledge platform corresponding to the target knowledge domain. Then, multiple knowledge data groups are extracted from the obtained target knowledge content, each including question information under the target knowledge domain and corresponding response information. Next, based on the extracted question information, response information, and preset question types in each knowledge data group, question-answer pairs are generated for each knowledge data group. Each question-answer pair includes a corresponding question and answer, where the question is generated based on the question information and preset question type in the knowledge data group, and the answer is generated based on the corresponding response and preset question type in that knowledge data group. Finally, an evaluation dataset is constructed based on the generated question-answer pairs for evaluating the target model.
[0017] This embodiment of the disclosure uses a knowledge platform corresponding to the target knowledge domain as the knowledge source. It constructs an evaluation dataset by extracting question information and its corresponding response information from the target knowledge domain, and then regenerating question-answer pairs based on the extracted information and preset question types. Since the evaluation dataset of this embodiment is constructed based on real data sources from the target knowledge domain, evaluating the model using this dataset can reflect the model's ability to handle complex real-world problems, thus improving the accuracy of model evaluation. Attached Figure Description
[0018] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.
[0019] Figure 1 A flowchart of a data processing method provided in an embodiment of this disclosure; Figure 2 This is a schematic diagram of a data processing method provided in an embodiment of the present disclosure; Figure 3 This is a schematic diagram of the structure of a data processing apparatus provided in an embodiment of the present disclosure; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation
[0020] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.
[0021] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.
[0022] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.
[0023] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.
[0024] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".
[0025] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.
[0026] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained. Currently, model evaluation methods are somewhat inadequate in reflecting a model's ability to handle complex real-world problems. Due to the highly specialized and scattered nature of knowledge, and the relatively sparse long-tail knowledge in the model's pre-training data, there are no relevant technologies that can accurately reflect a model's ability to process long-tail knowledge.
[0027] In theory, a qualified model evaluation benchmark should not only encompass this type of long-tail knowledge but also ensure that the evaluation tasks are designed to be rooted in real-world user application scenarios. However, the data sources for related model evaluation methods lack authenticity and diversity, making it difficult to reflect the diversity, depth, and dynamism of knowledge in the real world, especially in highly specialized vertical fields. Furthermore, related technologies typically rely on human experts to design and review evaluation questions, which is not only costly and time-consuming but also difficult to scale. In summary, due to the inability to provide high-quality evaluation datasets, the accuracy of model evaluation cannot be guaranteed.
[0028] To this end, this disclosure provides a data processing method. First, target knowledge content is obtained from at least one knowledge platform corresponding to the target knowledge domain. Then, multiple knowledge data groups are extracted from the obtained target knowledge content, each including question information under the target knowledge domain and corresponding response information. Next, based on the extracted question information, response information, and preset question types in each knowledge data group, question-answer pairs are generated for each knowledge data group. Each question-answer pair includes a corresponding question and answer, where the question is generated based on the question information and preset question type in the knowledge data group, and the answer is generated based on the corresponding response and preset question type in that knowledge data group. Finally, an evaluation dataset is constructed based on the generated question-answer pairs for evaluating the target model.
[0029] This embodiment of the disclosure uses a knowledge platform corresponding to the target knowledge domain as the knowledge source. It constructs an evaluation dataset by extracting question information and its corresponding response information from the target knowledge domain, and then regenerating question-answer pairs based on the extracted information and preset question types. Since the evaluation dataset of this embodiment is constructed based on real data sources from the target knowledge domain, evaluating the model using this dataset can reflect the model's ability to handle complex real-world problems. For example, it can reflect the model's ability to handle highly specialized, fragmented, and relatively sparse long-tail knowledge in the model's pre-training data, thus improving the accuracy of model evaluation.
[0030] For ease of understanding, embodiments of this disclosure provide a data processing method, which can be executed by a data processing device, wherein the device can be implemented in software and / or hardware, and is generally integrated into an electronic device. Reference Figure 1 The above is a flowchart of a data processing method provided in an embodiment of this disclosure, wherein the method includes: S101: Obtain the target knowledge content from at least one knowledge platform corresponding to the target knowledge domain.
[0031] In this embodiment of the disclosure, the target knowledge domain is one or more pre-determined knowledge domains, such as one or more highly specialized, scattered knowledge domains that are relatively sparse in the model's pre-training data.
[0032] In practical applications, before evaluating the target model, the knowledge domain for evaluation should be determined first, that is, the target model's ability to process a specific knowledge domain should be known through model evaluation.
[0033] In addition, embodiments of this disclosure can also obtain platform identifiers of one or more knowledge platforms corresponding to the target knowledge domain, which can be used as data sources for the target knowledge domain. Then, based on each platform identifier, page identifiers of at least one knowledge page that meets preset conditions are obtained from the knowledge platforms corresponding to the target knowledge domain. These preset conditions may include the page containing knowledge content of the target knowledge domain, and the time information and / or user interaction volume of the knowledge content meeting certain criteria. For example, the time information may meet a preset time range, and the user interaction volume may reach a preset interaction volume threshold.
[0034] In one optional implementation, user interaction volume can be any one of the following: number of replies, number of reads, or number of votes. Meeting the time information standard for knowledge content means that the time information of the knowledge content is within a preset time range, used to control the timeliness of the acquired target knowledge content. Knowledge pages under the knowledge platform corresponding to the target knowledge domain are filtered based on user interaction volume and time information (such as the knowledge content's publication time) to determine the knowledge pages that meet both the user interaction volume and time information standards, which are then used to acquire the target knowledge content.
[0035] After obtaining the page identifier of at least one knowledge page that meets the preset conditions, the target knowledge content is retrieved from the corresponding knowledge page based on the obtained page identifier. In one optional implementation, the target knowledge content can be obtained by taking screenshots of the pages. Specifically, screenshots are taken of each knowledge page according to the page identifiers to obtain multiple knowledge content screenshots, and each knowledge content screenshot is used as the target knowledge content.
[0036] In one optional implementation, the steps for acquiring target knowledge content can be executed concurrently for different knowledge pages on different knowledge platforms, so as to efficiently acquire target knowledge content in the target knowledge domain.
[0037] The target knowledge content in this embodiment may include multimodal information, such as images, text, charts, code snippets, formulas, and diagrams. All of the above modal information plays an important role in the exchange of professional knowledge.
[0038] S102: Extract multiple knowledge data groups from the target knowledge content.
[0039] The knowledge data group includes question information under the target knowledge domain and at least one response information corresponding to the question information.
[0040] In this embodiment of the disclosure, after obtaining the target knowledge content, question information and corresponding response information under the target knowledge domain are extracted from the target knowledge content. The same question information may correspond to one or more response information, which are used to construct a knowledge data group.
[0041] In one optional implementation, if the target knowledge content includes screenshots of knowledge content, then by performing content recognition and semantic analysis on each screenshot of knowledge content, question information under the target knowledge domain and the corresponding response information are extracted from each screenshot of knowledge content to construct a knowledge data group.
[0042] In another optional implementation, if the target knowledge content includes content in the form of posts, the title, question description and related reply content can be extracted from the posts, and then the question information under the target knowledge domain and the corresponding reply information can be further extracted from the extracted content to construct a knowledge data group.
[0043] In one optional implementation, after acquiring the target knowledge content, not only can question information and corresponding response information within the target knowledge domain be extracted from the target knowledge content, but also contextual information corresponding to the question information can be extracted. The contextual information is typically extracted from the contextual location corresponding to the question information and is used to aid in understanding the question information and its corresponding response information. For example, embodiments of this disclosure can extract a post from a blogger asking a question within the target knowledge domain, responses from others to that post, and other comment information related to that post. In this embodiment of the disclosure, multiple knowledge data sets can be extracted from the target knowledge content obtained from the knowledge platform corresponding to the target knowledge domain. Each knowledge data set includes a triple consisting of a question from the target knowledge domain, at least one corresponding response, and contextual information related to the question. In other words, this embodiment of the disclosure can extract multiple triples from the target knowledge domain for subsequent construction of an evaluation dataset.
[0044] In this embodiment of the disclosure, the question information and response information included in the knowledge data group extracted from the target knowledge content have undergone validity testing. Validity testing may include checking the clarity of the extracted question information, i.e., whether the question description is clear. Additionally, validity testing may also include checking whether the extracted question information and response content contain substantive content. For example, it may check whether the extracted question information contains valid response information. If it only contains question information but no response information, it fails the validity test; if the response information is meaningless characters, it fails the validity test; or a simple popular science post that does not contain question information will also fail the validity test.
[0045] In addition, the answers to the questions in the knowledge data group of this disclosure are unique and verifiable, and can be verified through reasoning or experimentation, thus eliminating questions with open boundaries, multiple possible answers, or paths.
[0046] S103: Based on the question information and response information in the knowledge data group and the preset question type, generate the question-answer pair content corresponding to the knowledge data group.
[0047] The question-and-answer pair includes corresponding question content and answer content. The question content is generated based on the question information in the knowledge data group and the preset question type, and the answer content is generated based on the reply content and the preset question type.
[0048] In this embodiment of the disclosure, after extracting multiple knowledge data groups of the target knowledge domain, question-answer pairs corresponding to each knowledge data group are generated based on the question information and response information in each knowledge data group and the preset question type.
[0049] In one optional implementation, before generating the question-and-answer pairs corresponding to each knowledge data group, semantic deduplication can be performed on each knowledge data group. Specifically, the question information in the extracted knowledge data groups is converted into semantic vectors. Then, based on the semantic vectors corresponding to the question information in each knowledge data group, semantic deduplication is performed on the extracted knowledge data groups, thereby reducing the semantic similarity probability of the knowledge data groups used to generate the question-and-answer pairs.
[0050] In one optional implementation, a pre-trained semantic similarity model can be used to convert question information in each knowledge data group into semantic vectors, and each semantic vector is added to a pre-built semantic vector library. This semantic vector library stores the semantic vectors corresponding to the question information in each knowledge data group. The similarity of each semantic vector added to the library is compared with the semantic vectors already stored in the library. If the similarity between the newly added semantic vector and any of the semantic vectors already stored in the library is greater than a preset similarity threshold, the added semantic vector is considered semantically redundant, and the knowledge data group to which the question information belongs is marked as "unqualified," with the reason for unqualification updated for subsequent adjustments to the knowledge data group. If the similarity between the newly added semantic vector and any of the semantic vectors already stored in the library is not greater than the preset similarity threshold, the knowledge data group to which the question information belongs is marked as "qualified" and can be used for subsequent question-answer pair generation.
[0051] like Figure 2 The diagram illustrates a data processing procedure according to an embodiment of this disclosure. After obtaining the target knowledge content based on the platform identifier of the knowledge platform in the target knowledge domain, multiple knowledge data groups are extracted from the target knowledge content, such as... Figure 2 The knowledge data groups 1 and 2 shown are processed by semantic deduplication and validity checks. For the knowledge data groups that pass the semantic deduplication and validity checks, question-answer pairs are generated for each knowledge data group, such as... Figure 2 The question-and-answer pair content 1 and question-and-answer pair content 2.
[0052] In practical applications, various question types can be pre-set before generating question-and-answer pairs, such as single-choice questions, multiple-choice questions, and short-answer questions. Different methods can be used to generate question-and-answer pairs that meet the requirements of different pre-set question types.
[0053] Assuming multiple question types are preset, in order to ensure a balanced distribution of the generated question data of various question types, this embodiment of the disclosure can divide the extracted knowledge data group of the target knowledge domain into multiple sets corresponding to the types of question types, and use the knowledge data group in each set to generate question-answer pairs of the preset question types corresponding to that set.
[0054] The question-and-answer pairs generated by each knowledge data group in this embodiment include corresponding question content and answer content. Specifically, the question information in the knowledge data group is used to generate question content, and the response information in the knowledge data group is used to generate answer content. In addition, in the process of generating question content and answer content, the characteristics of the preset question type need to be considered. For example, according to the characteristics of multiple-choice questions, multiple options need to be generated, including correct options and incorrect options.
[0055] In one optional implementation, since the knowledge data set may include question information, response information, and context information, this embodiment of the disclosure can generate question-and-answer pair content corresponding to the knowledge data set based on the question information, response information, and context information in the knowledge data set. The question-and-answer pair content includes not only the question content and the answer content, but also the reasoning and understanding content corresponding to the question content, used to explain the reasoning process from the question content to the corresponding answer content. Specifically, context information can be used to generate the reasoning and understanding content in the question-and-answer pair content.
[0056] In one optional implementation, the extracted knowledge data groups and preset question types can be input into a large language model, which then generates question-answer pairs corresponding to each knowledge data group. Specifically, the large language model generates question content based on the question information and preset question types in the knowledge data group, generates answer content for the question based on the response information and preset question types, and generates reasoning and understanding content based on the question information, response information, contextual content, and preset reminders.
[0057] In practical applications, when the preset question type is multiple choice, multiple choice questions and multiple options are generated based on the question information and response information in each knowledge data group, constituting the question content; among these multiple options, at least one is correct. Specifically, when generating incorrect options, it is necessary to consider not only the question information but also the response information.
[0058] Then, based on the response information in the knowledge data set, the answer content corresponding to the question is generated, and a question-answer pair is constructed based on the answer content and the question content. Furthermore, based on the contextual information in each knowledge data set, inference explanations from the question content to the answer content can be generated to explain why one or more options were chosen.
[0059] When the preset question types include short-answer questions, the system generates question content based on the question information in each knowledge data group, and generates answer content and key points based on the response information in that knowledge data group. Since short-answer questions require answers that do not need to be completely identical to the answer key, but rather satisfy the corresponding key points, points are used to determine the accuracy of the answer to that question. Typically, each question contains 1-5 core key points, which are considered key points.
[0060] After generating answer content, question content, and test point content based on each knowledge data group, the question-answer pair content corresponding to that knowledge data group is constructed based on the answer content, question content, and test point content corresponding to the same knowledge data group.
[0061] S104: Construct an evaluation dataset based on the content of the question-and-answer pairs.
[0062] The evaluation dataset is used to evaluate the target model.
[0063] In this embodiment of the disclosure, after generating the question-and-answer pairs corresponding to each knowledge data group, each question-and-answer pair is used as an evaluation data sample to construct an evaluation dataset.
[0064] like Figure 2 As shown, after generating multiple question-answer pairs such as Question-Answer Pair 1 and Question-Answer Pair 2, an evaluation dataset is constructed based on each question-answer pair to evaluate the capabilities of the target model.
[0065] In one optional implementation, to ensure that the difficulty of the questions in each evaluation data sample in the evaluation dataset is within an appropriate range, the difficulty of the generated question-answer pairs can be assessed. Specifically, the question content from each evaluation data sample in the evaluation dataset can be input into a difficulty assessment model, and the difficulty of the question content in the evaluation data sample can be evaluated by the accuracy of the answer content output by the difficulty assessment model.
[0066] In practical applications, questions with scores higher than a preset first score and evaluation data samples with scores of a preset second score can be extracted from the evaluation dataset. This ensures that the difficulty of the questions in the remaining question-and-answer pairs in the evaluation dataset is within an appropriate range, making them more suitable for model evaluation.
[0067] In one optional implementation, after obtaining the platform identifier of at least one knowledge platform corresponding to the target knowledge domain, the data processing task corresponding to the data processing method provided in this embodiment of the present disclosure is triggered. During the execution of the data processing task, in order to improve the transparency of task execution, the data processing progress can be visualized and managed so that users can understand the task execution progress in real time and improve the user experience.
[0068] In one alternative implementation, version management can be implemented for the evaluation dataset, and quantitative analysis can be performed on different versions of the evaluation dataset to improve the scientific nature and systematicity of the evaluation iteration.
[0069] In the data processing method provided in this embodiment, firstly, target knowledge content is obtained from at least one knowledge platform corresponding to the target knowledge domain. Then, multiple knowledge data groups are extracted from the obtained target knowledge content, each including question information under the target knowledge domain and corresponding response information. Next, based on the extracted question information, response information, and preset question types in each knowledge data group, question-answer pairs are generated for each knowledge data group. Each question-answer pair includes a corresponding question and answer, where the question is generated based on the question information and preset question type in the knowledge data group, and the answer is generated based on the corresponding response and preset question type in that knowledge data group. Finally, an evaluation dataset is constructed based on the generated question-answer pairs for evaluating the target model.
[0070] Since the evaluation dataset constructed in this embodiment is generated based on raw data dynamically obtained from a real knowledge exchange platform in the target knowledge domain, the evaluation dataset used in this embodiment can reflect the model's ability to handle complex real-world problems and improve the accuracy of model evaluation.
[0071] To implement the above embodiments, this disclosure also proposes a data processing apparatus. Figure 3 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this disclosure. The device can be implemented by software and / or hardware and is generally integrated into an electronic device. Figure 3 As shown, the device includes: The acquisition module 301 is used to acquire target knowledge content from at least one knowledge platform corresponding to the target knowledge domain. Extraction module 302 is used to extract multiple knowledge data groups from the target knowledge content; wherein, the knowledge data group includes question information under the target knowledge domain and at least one response information corresponding to the question information; The generation module 303 is used to generate question-answer pair content corresponding to the knowledge data group based on the question information and reply information in the knowledge data group and the preset question type; wherein, the question-answer pair content includes question content and answer content with corresponding relationship, the question content is generated based on the question information in the knowledge data group and the preset question type, and the answer content is generated based on the reply content and the preset question type; The construction module 304 is used to construct an evaluation dataset based on the question-and-answer pair content; wherein the evaluation dataset is used to evaluate the target model.
[0072] In one optional implementation, the knowledge data set further includes context information corresponding to the question information, and the generation module is specifically used for: Based on the question information, response information, context information, and preset question types in the knowledge data set, question-answer pair content corresponding to the knowledge data set is generated; wherein, the question-answer pair content also includes reasoning explanation content corresponding to the question content, and the reasoning explanation content is used to explain the reasoning process from the question content to the corresponding answer content.
[0073] In one optional embodiment, the apparatus further includes: The vector conversion module is used to convert the question information in the multiple knowledge data groups into semantic vectors respectively; The semantic deduplication module is used to perform semantic deduplication on the multiple knowledge data groups based on the semantic vectors corresponding to the question information in the multiple knowledge data groups.
[0074] In one optional implementation, the generation module includes: The first generation submodule is used to generate multiple-choice questions and multiple options based on the question information and response information in the knowledge data group when the preset question type includes multiple-choice questions, thereby constituting the question content; wherein, the multiple options include at least one correct option; The first construction submodule is used to generate the answer content corresponding to the question content based on the reply information in the knowledge data group, and to construct the question-answer pair content based on the answer content and the question content.
[0075] In one optional implementation, the generation module includes: The second generation submodule is used to generate question content based on the question information in the knowledge data group when the preset question types include short answer questions, and to generate answer content and test point content based on the response information in the knowledge data group; wherein, the test point content is used to determine the accuracy of the answer to the question content; The second construction submodule is used to construct question-answer pairs based on the answer content, the question content, and the test point content.
[0076] In one optional implementation, the acquisition module includes: The first acquisition submodule is used to acquire the page identifier of at least one knowledge page that meets preset conditions from at least one knowledge platform corresponding to the target knowledge domain. The second acquisition submodule is used to acquire target knowledge content from the knowledge page based on the page identifier.
[0077] In one optional implementation, the second acquisition submodule includes: The screenshot submodule is used to take screenshots of the at least one knowledge page according to the page identifier to obtain multiple knowledge content images; The extraction module is specifically used for: Multiple knowledge data sets are obtained by performing content recognition and semantic analysis on the multiple knowledge content images.
[0078] The data processing apparatus provided in this embodiment first obtains target knowledge content from at least one knowledge platform corresponding to the target knowledge domain. Then, it extracts multiple knowledge data groups from the obtained target knowledge content, each including question information under the target knowledge domain and corresponding response information. Next, based on the extracted question information, response information, and preset question types in each knowledge data group, it generates question-answer pairs corresponding to each knowledge data group. Each question-answer pair includes a corresponding question and answer, where the question is generated based on the question information and preset question type in the knowledge data group, and the answer is generated based on the corresponding response and preset question type in that knowledge data group. Finally, it constructs an evaluation dataset based on the generated question-answer pairs for evaluating the target model.
[0079] This embodiment of the disclosure uses a knowledge platform corresponding to the target knowledge domain as the knowledge source. It constructs an evaluation dataset by extracting question information and its corresponding response information from the target knowledge domain, and then regenerating question-answer pairs based on the extracted information and preset question types. Since the evaluation dataset of this embodiment is constructed based on real data sources from the target knowledge domain, evaluating the model using this dataset can reflect the model's ability to handle complex real-world problems, thus improving the accuracy of model evaluation.
[0080] In addition to the methods and apparatus described above, this disclosure also provides a computer-readable storage medium storing instructions that, when executed on a terminal device, cause the terminal device to implement the data processing method described in this disclosure.
[0081] This disclosure also provides a computer program product, which includes a computer program / instructions that, when executed by a processor, implement the data processing method described in this disclosure.
[0082] In addition, this disclosure also provides an electronic device, see [link to relevant documentation]. Figure 4As shown, it may include: The electronic device includes a processor 401, a memory 402, an input device 403, and an output device 404. The number of processors 401 in the electronic device can be one or more. Figure 4 Taking a processor as an example. In some embodiments of this disclosure, the processor 401, memory 402, input device 403, and output device 404 can be connected via a bus or other means, wherein, Figure 4 Taking the example of a connection between China and Israel via a bus.
[0083] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing of the electronic device by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function, etc. In addition, the memory 402 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. The input device 403 can be used to receive input digital or character information, and to generate signal inputs related to user settings and function control of the electronic device.
[0084] Specifically in this embodiment, the processor 401 loads the executable files corresponding to the processes of one or more applications into the memory 402 according to the following instructions, and the processor 401 runs the applications stored in the memory 402 to realize the various functions of the above-mentioned electronic device.
[0085] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0086] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A data processing method, comprising: obtaining target knowledge content from at least one knowledge platform corresponding to a target knowledge field; extracting a plurality of knowledge data groups from the target knowledge content, wherein the knowledge data groups comprise question information under the target knowledge field, and at least one reply information corresponding to the question information; generating question and answer pair content corresponding to the knowledge data groups based on the question information and the reply information in the knowledge data groups and a preset question type, wherein the question and answer pair content comprises question content and answer content having a corresponding relationship, the question content is generated based on the question information in the knowledge data groups and the preset question type, and the answer content is generated based on the reply content and the preset question type; constructing an evaluation data set based on the question and answer pair content, wherein the evaluation data set is used to evaluate a target model.
2. The data processing method of claim 1, wherein the knowledge data groups further comprise context information corresponding to the question information, and the generating of the question and answer pair content corresponding to the knowledge data groups based on the question information, the reply information, the context information, and the preset question type comprises: generating the question and answer pair content corresponding to the knowledge data groups based on the question information, the reply information, the context information, and the preset question type, wherein the question and answer pair content further comprises reasoning explanation content corresponding to the question content, and the reasoning explanation content is used to explain an inference process from the question content to the corresponding answer content.
3. The data processing method of claim 1, wherein before the generating of the question and answer pair content corresponding to the knowledge data groups based on the question information and the reply information in the knowledge data groups and the preset question type, the method further comprises: converting the question information in the plurality of knowledge data groups into semantic vectors, respectively; performing semantic deduplication on the plurality of knowledge data groups according to the semantic vectors corresponding to the question information in the plurality of knowledge data groups, respectively.
4. The data processing method of claim 1, wherein the generating of the question and answer pair content corresponding to the knowledge data groups based on the question information and the reply information in the knowledge data groups and the preset question type comprises: in a case where the preset question type comprises a selection question type, generating a selection question and a plurality of options based on the question information and the reply information in the knowledge data groups to constitute the question content, wherein the plurality of options comprise at least one correct option; generating answer content corresponding to the question content based on the reply information in the knowledge data groups, and constructing the question and answer pair content based on the answer content and the question content.
5. The data processing method of claim 1, wherein the generating of the question and answer pair content corresponding to the knowledge data groups based on the question information and the reply information in the knowledge data groups and the preset question type comprises: in a case where the preset question type comprises a short answer question type, generating question content based on the question information in the knowledge data groups, and generating answer content and a focus point content based on the reply information in the knowledge data groups, wherein the focus point content is used to determine an accuracy degree of an answer to the question content. Construct a question and answer pair content based on the answer content, the question content and the exam point content. 6.The data processing method of claim 1, wherein the obtaining of the target knowledge content from at least one knowledge platform corresponding to the target knowledge field comprises: obtaining, from at least one knowledge platform corresponding to the target knowledge field, a page identifier of at least one knowledge page satisfying a preset condition; obtaining the target knowledge content from the knowledge page according to the page identifier. 7.The data processing method of claim 6, wherein the obtaining of the target knowledge content from the knowledge page according to the page identifier comprises: respectively taking screenshots of the at least one knowledge page according to the page identifier to obtain a plurality of knowledge content images; and the extracting of the plurality of knowledge data groups from the target knowledge content comprises: performing content recognition and semantic analysis processing on the plurality of knowledge content images to obtain the plurality of knowledge data groups. 8.A data processing apparatus, comprising: an obtaining module configured to obtain target knowledge content from at least one knowledge platform corresponding to a target knowledge field; an extracting module configured to extract a plurality of knowledge data groups from the target knowledge content, wherein the knowledge data groups include question information under the target knowledge field and at least one reply information corresponding to the question information; a generating module configured to generate question and answer pair content corresponding to the knowledge data groups based on the question information and the reply information in the knowledge data groups and a preset question type, wherein the question and answer pair content includes question content and answer content having a corresponding relationship, the question content is generated based on the question information in the knowledge data groups and the preset question type, and the answer content is generated based on the reply content and the preset question type; a constructing module configured to construct an evaluation data set based on the question and answer pair content, wherein the evaluation data set is used to evaluate a target model. 9.An electronic device, comprising: a processor; a memory for storing executable instructions of the processor; the processor is configured to read the executable instructions from the memory and execute the instructions to implement the method of any one of claims 1-7. 10.A computer readable storage medium, the storage medium storing a computer program, the computer program being used to execute the method of any one of claims 1-7. 11.A computer program product, the computer program product comprising computer programs / instructions, the computer programs / instructions being executed by a processor to implement the method of any one of claims 1-7.