Data collection method and device, equipment and medium

Abstract

The invention discloses a data collection method, device and equipment and a medium, relates to the technical field of computers, is applied to an intelligent agent, and comprises the following steps: determining a target theme field corresponding to a data collection demand and a required data volume by using a few-sample learning algorithm; determining an initial seed word list corresponding to the target subject field from a seed word manager, and expanding the initial seed word list by using a target pre-training model; identifying the part-of-speech of each target seed word in the expanded seed word list based on a least-to-most prompt algorithm, and generating prompt information corresponding to the target seed word by using a prompt information generation template corresponding to the part-of-speech of the target seed word; based on the prompt information, a target data set corresponding to the needed data size is obtained in a network retrieval mode, and the output of the current step serves as the input of the next step through a thinking chain algorithm. The thinking chain algorithm forms a coherent thinking process, and the quality and quantity of the collected data can be guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data collection method, device, equipment and medium. Background Art

[0002] Pre-trained models cannot be separated from high-quality training data during training. Training data is the basis of model learning, and its quality and diversity are directly related to the performance of the model. High-quality and diverse training data can help the model better learn the rules and features in the data, thereby improving the accuracy and generalization ability of the model. At the same time, through reasonable data preprocessing and screening, the amount of data can be effectively reduced and the quality of the data set can be improved, which is crucial to improving training efficiency and model performance. Therefore, in the training of pre-trained models, obtaining and preparing high-quality training data is an indispensable key step.

[0003] At present, the data required for the model pre-training and model fine-tuning stages are mainly obtained through the data accumulated by the manufacturer itself or the data obtained by crawling from public data. The data accumulated by the manufacturer itself is very valuable and strictly controlled. The data obtained by crawling from public data is of lower quality, and the number is also small after filtering and other operations on the crawler data.

[0004] It can be seen that how to ensure the quality and quantity of the collected data is a problem that technical personnel in this field need to solve. Summary of the invention

[0005] The purpose of the embodiment of the present invention is to provide a data collection method, device, equipment and medium. The specific scheme is as follows:

[0006] In a first aspect, the present invention discloses a data collection method, applied to an intelligent agent, comprising:

[0007] Use the few-shot learning algorithm to determine the target subject area and the amount of data required for data collection, and use the thought chain algorithm to output the target subject area;

[0008] Determine an initial seed word list corresponding to the target subject area outputted by the thought chain algorithm from the seed word manager, expand the initial seed word list by using the target pre-trained model, and output the expanded seed word list by using the thought chain algorithm;

[0009] Based on the least-to-most hint algorithm, the part of speech of each target seed word in the expanded seed word list output by the thought chain algorithm is identified, and a hint information generation template corresponding to the part of speech of the target seed word is used to generate hint information corresponding to the target seed word, and the hint information is output by the thought chain algorithm;

[0010] Based on the prompt information output by the thought chain algorithm and using network retrieval method, the target data set corresponding to the required data volume is obtained.

[0011] Optionally, using a prompt information generation template corresponding to the part of speech of the target seed word to generate prompt information corresponding to the target seed word includes:

[0012] Determine whether the part of speech of the target seed word belongs to the preset part of speech category;

[0013] If the part of speech of the target seed word belongs to the preset part of speech category, then using the general prompt information generation template to generate general prompt information corresponding to the target seed word;

[0014] If the part of speech of the target seed word does not belong to the preset part of speech category, then using the exclusive prompt information generation template to generate exclusive prompt information corresponding to the target seed word;

[0015] Accordingly, the thought chain algorithm is used to output the prompt information, including:

[0016] The thought chain algorithm is used to output general prompt information and / or exclusive prompt information.

[0017] Optionally, based on the prompt information output by the thought chain algorithm and using a network search method to obtain a target data set corresponding to the required data volume, including:

[0018] Conduct network search on the general prompt information output by the thought chain algorithm to obtain network search content;

[0019] Determine paragraph-level search content and non-paragraph-level search content from the network search content, and determine the paragraph-level search content as a first data set;

[0020] Inputting the exclusive prompt information output by the thought chain algorithm and / or the prompt information corresponding to the non-paragraph level search content into the target pre-training model, so as to obtain a second data set output by the target pre-training model;

[0021] A target data set is generated including a first data set and a second data set.

[0022] Optionally, generating a target data set including the first data set and the second data set includes:

[0023] Screening out a third data set corresponding to the target subject area from the historical data sets stored in the database, and generating a target data set including the first data set, the second data set and the third data set;

[0024] Accordingly, after generating the target data set including the first data set, the second data set and the third data set, the method further includes:

[0025] The target scoring model is used to perform a quality score on the target dataset to filter out the target sub-datasets whose quality scores are greater than a preset threshold, and the target sub-datasets are stored in the database.

[0026] Optionally, the initial seed word list is expanded using the target pre-trained model, including:

[0027] Using the target pre-training model and the expansion prompt information generation template, each initial seed word in the initial seed word list is expanded to obtain each candidate seed word; wherein the initial seed word is a preset representative word under the target subject area;

[0028] Filtering each candidate seed word based on a preset filtering condition to obtain each filtered seed word, and aggregating each filtered seed word to obtain each aggregated seed word;

[0029] An expanded seed word list including the initial seed words and the aggregated seed words is constructed.

[0030] Optionally, use a few-shot learning algorithm to determine the target subject area and amount of data required for data collection needs, including:

[0031] Receive the current data collection instruction through a preset interface, and extract the data collection requirements in the current data collection instruction using a few-sample learning algorithm;

[0032] Filter out target subject areas corresponding to data collection requirements from various preset subject areas built into the seed word manager, and determine the required amount of data corresponding to the data collection requirements.

[0033] Optionally, use a few-shot learning algorithm to determine the target subject area and amount of data required for data collection needs, including:

[0034] Extracting a first data collection requirement in a current data collection instruction using a few-sample learning algorithm;

[0035] If the first data collection requirement meets the preset extraction success condition, the first subject area and the first data volume in the first data collection requirement are determined as the target subject area and the required data volume respectively;

[0036] If the first data collection requirement does not meet the preset extraction success condition, then use a regular expression to extract the second data collection requirement in the current data collection instruction;

[0037] If the second data collection requirement meets the preset extraction success condition, the second subject area and the second data volume in the second data collection requirement are determined as the target subject area and the required data volume respectively;

[0038] If the second data collection requirement does not meet the preset extraction success conditions, the target pre-trained model is used to extract the third data collection requirement in the current data collection instruction, and the third subject area and the third data amount in the third data collection requirement are respectively determined as the target subject area and the required data amount.

[0039] In a second aspect, the present invention discloses a data collection device, applied to an intelligent agent, comprising:

[0040] The demand determination module is used to determine the target subject area and the required data volume corresponding to the data collection demand using the few-sample learning algorithm, and output the target subject area using the thought chain algorithm;

[0041] A seed word expansion module is used to determine an initial seed word list corresponding to the target subject area output by the thought chain algorithm from the seed word manager, expand the initial seed word list by using the target pre-trained model, and output the expanded seed word list by using the thought chain algorithm;

[0042] a prompt generation module, for identifying the part of speech of each target seed word in the expanded seed word list output by the thought chain algorithm based on the least-to-most prompt algorithm, and generating prompt information corresponding to the target seed word by using a prompt information generation template corresponding to the part of speech of the target seed word, and outputting the prompt information by using the thought chain algorithm;

[0043] The data set acquisition module is used to acquire the target data set corresponding to the required data volume based on the prompt information output by the thought chain algorithm and using the network retrieval method.

[0044] In a third aspect, the present invention discloses an electronic device, comprising:

[0045] Memory, used to store computer programs;

[0046] The processor is used to execute the computer program to implement the steps of the aforementioned disclosed data collection method.

[0047] In a fourth aspect, the present invention discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the steps of the aforementioned disclosed data collection method are implemented.

[0048] It can be seen that the present invention is applied to an intelligent body, including: using a few-sample learning algorithm to determine the target subject area and the required data volume corresponding to the data collection needs, and using a thinking chain algorithm to output the target subject area; determining an initial seed word list corresponding to the target subject area output by the thinking chain algorithm from a seed word manager, and using a target pre-training model to expand the initial seed word list, and using the thinking chain algorithm to output the expanded seed word list; based on the least to most prompt algorithm, identifying the part of speech of each target seed word in the expanded seed word list output by the thinking chain algorithm, and using a prompt information generation template corresponding to the part of speech of the target seed word to generate prompt information corresponding to the target seed word, and using the thinking chain algorithm to output the prompt information; based on the prompt information output by the thinking chain algorithm and using a network retrieval method, obtaining a target data set corresponding to the required data volume.

[0049] The beneficial effects are as follows: the present application is applied to an intelligent agent, and a few-sample learning algorithm is used to determine the target subject area and the required amount of data for each data collection requirement, so as to correctly extract the user's data collection requirements, and expand the initial seed list under the target subject area to obtain an expanded list containing sufficient target seeds, that is, to improve the seed diversity and coverage. Subsequently, based on the minimum to maximum prompt algorithm, a prompt information generation template corresponding to the part of speech of the target seed is used to generate prompt information corresponding to the target seed. On the one hand, the part of speech of simple target seeds can be preferentially identified to maintain high efficiency and accuracy, and on the other hand, the prompt information is more comprehensive and rich, so that based on the prompt information, the prompt information can be collected from the target pre-training model, Data collection is completed in two aspects: network retrieval. Network retrieval is to obtain corresponding data from public data, and the target pre-training model generation method can generate new relevant data based on existing data. The two-aspect data collection can ensure both the quality and quantity of the data. Furthermore, in the process of controlling the data using the thinking chain algorithm, the reasoning result of each step will be used as the input of the next step. That is to say, the input in the list expansion step is the target subject area output by the previous step, the input in the prompt information generation step is the expanded seed word list output by the previous step, and the input of the data set acquisition step is the prompt information output by the previous step, thus forming a coherent thinking process. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0051] Figure 1A flow chart of a data collection method provided by an embodiment of the present invention;

[0052] Figure 2 A specific schematic diagram of collecting requirements and extracting them is provided in an embodiment of the present invention;

[0053] Figure 3 A specific schematic diagram of expanding a seed word list provided by an embodiment of the present invention;

[0054] Figure 4 A specific target data set acquisition schematic diagram provided by an embodiment of the present invention;

[0055] Figure 5 A specific data collection method flow chart provided by an embodiment of the present invention;

[0056] Figure 6 A schematic diagram of a specific intelligent agent structure provided by an embodiment of the present invention;

[0057] Figure 7 A specific data collection schematic diagram provided for an embodiment of the present invention;

[0058] Figure 8 A schematic diagram of the structure of a data collection device provided by an embodiment of the present invention;

[0059] Fig. 9 A structural diagram of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0060] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] Pre-trained models cannot be separated from high-quality training data during training. Training data is the basis of model learning, and its quality and diversity are directly related to the performance of the model. High-quality and diverse training data can help the model better learn the patterns and features in the data, thereby improving the accuracy and generalization ability of the model. At the same time, through reasonable data preprocessing and screening, the amount of data can be effectively reduced and the quality of the data set can be improved, which is crucial to improving training efficiency and model performance. Therefore, in the training of pre-trained models, obtaining and preparing high-quality training data is an indispensable key step.

[0062] At present, the data required for the model pre-training and model fine-tuning stages are mainly obtained through the data accumulated by the manufacturer itself or the data obtained by crawling from public data. The data accumulated by the manufacturer itself is very valuable and strictly controlled. The data obtained by crawling from public data is of lower quality, and the number is also small after filtering and other operations on the crawler data.

[0063] The terms "including" and "having" in the specification of the present invention and the above-mentioned drawings, as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but may include steps or units that are not listed.

[0064] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0065] Next, a data collection solution provided by an embodiment of the present invention is described in detail. Figure 1 A data collection method provided in an embodiment of the present invention is applied to an intelligent agent, comprising:

[0066] Step S11: Use a few-sample learning algorithm to determine the target subject area and the required data volume corresponding to the data collection requirements, and use a thought chain algorithm to output the target subject area.

[0067] As Large Language Models (LLMs) continue to have a profound impact on all walks of life, more and more manufacturers are investing their energy in the pre-training and fine-tuning of LLMs. Regardless of the pre-training or fine-tuning stage, the acquisition and quality of data sets are extremely important. Especially in the pre-training stage, LLMs require a large amount of data sets, but the amount of open source data sets cannot meet LLM's demand for data sets.

[0068] With the continuous evolution of artificial intelligence, the concept of agent has been proposed. Agent refers to an agent that can perceive the environment and take actions to achieve specific goals. It has autonomy, adaptability and interaction capabilities. The agent perceives changes in the environment, makes judgments and decisions based on the knowledge and algorithms it has learned, and then performs actions to affect the environment or achieve predetermined goals. The core of the agent is the ability to learn autonomously and continuously evolve in order to better complete tasks and adapt to complex environments.

[0069] In a specific embodiment, a few-sample learning algorithm is used to determine the target subject area and the required data volume corresponding to the data collection requirements, including: receiving the current data collection instruction through a preset interface, and extracting the data collection requirements in the current data collection instruction using the few-sample learning algorithm; screening out the target subject area corresponding to the data collection requirements from various preset subject areas built into the seed word manager, and determining the required data volume corresponding to the data collection requirements.

[0070] The intelligent agent can receive the current data collection instruction through a preset interface, wherein the preset interface can be a voice instruction receiving interface, a text instruction receiving interface, a picture instruction receiving interface, or a trigger instruction receiving interface. When the preset interface is specifically a trigger instruction receiving interface, the user can directly select various selectable trigger instruction receiving interfaces on the intelligent agent display interface, for example, trigger instruction receiving interface a corresponds to subject area 1, trigger instruction receiving interface b corresponds to preset subject area 2, trigger instruction receiving interface c corresponds to preset subject area 3, trigger instruction receiving interface 1 corresponds to preset data volume 1, trigger instruction receiving interface 2 corresponds to preset data volume 2, and trigger instruction receiving interface 3 corresponds to preset data volume 3. When the user selects trigger instruction receiving interface 2 and trigger instruction receiving interface 3, the intelligent agent can obtain the current data collection instruction including preset subject area 2 and preset data volume 3 through the trigger instruction receiving interface. The intelligent agent provides a variety of preset interfaces, especially the trigger instruction receiving interface, so that the user can directly select the corresponding trigger instruction on the intelligent agent display interface according to their needs and preferences. This intuitive and easy-to-use interactive method greatly simplifies the operation process and lowers the usage threshold, so that even non-professionals can easily get started.

[0071] It is understandable that the current data collection instruction contains at least two types of information: subject area and data volume. In this way, after the agent extracts the data collection requirements in the current data collection instruction, it can filter out the target subject area corresponding to the data collection requirements from the various preset subject areas built into the seed word manager, and determine the required data volume corresponding to the data collection requirements. Among them, the various preset subject areas built into the seed word manager are, for example, culture, education, life, science, history, art, religion, humanities, nature and other fields.

[0072] In this embodiment, a few-sample learning algorithm is used to determine the target subject area and required data volume corresponding to the data collection requirement, including: using the few-sample learning algorithm to extract the first data collection requirement in the current data collection instruction; if the first data collection requirement meets the preset extraction success condition, the first subject area and the first data volume in the first data collection requirement are respectively determined as the target subject area and the required data volume; if the first data collection requirement does not meet the preset extraction success condition, a regular expression is used to extract the second data collection requirement in the current data collection instruction; if the second data collection requirement meets the preset extraction success condition, the second subject area and the second data volume in the second data collection requirement are respectively determined as the target subject area and the required data volume; if the second data collection requirement does not meet the preset extraction success condition, the target pre-trained model is used to extract the third data collection requirement in the current data collection instruction, and the third subject area and the third data volume in the third data collection requirement are respectively determined as the target subject area and the required data volume.

[0073] Few Shot learning is a few-sample learning algorithm that decomposes tasks using a small amount of sample data. That is, the FewShot algorithm is suitable for situations where the correct user needs cannot be extracted or the user needs cannot be met. In this embodiment, for example Figure 2 A specific collection requirement extraction schematic diagram is shown, in which the first data collection requirement in the current data collection instruction can be extracted using a few-sample learning algorithm. If the extraction fails, that is, the preset extraction success condition is not met, the second data collection requirement in the current data collection instruction can be extracted using a regular expression. If the extraction still fails, the third data collection requirement in the current data collection instruction can be extracted using a target pre-trained model, wherein the first data collection requirement and / or the second data collection requirement cannot be extracted, or the first data collection requirement and / or the second data collection requirement do not conform to the data collection logic, which are all considered to fail to meet the preset extraction success condition. It can be understood that if the first data collection requirement or the second data collection requirement or the third data collection requirement meets the preset extraction success condition, the subject area and data volume in the first data collection requirement or the second data collection requirement or the third data collection requirement are respectively determined as the target subject area and the required data volume.

[0074] In this embodiment, the target subject area is outputted using the thought chain algorithm, that is, after the target subject area is determined, the target subject area is the output of this step.

[0075] Step S12: Determine from the seed word manager an initial seed word list corresponding to the target subject area output by the thought chain algorithm, expand the initial seed word list by using the target pre-trained model, and output the expanded seed word list by using the thought chain algorithm.

[0076] It can be understood that the output of the previous step is the target subject area, and the input of this step is the target subject area. The initial seed word list corresponding to the target subject area contains various initial seed words. The initial seed words are preset representative words under the target subject area. For example, the initial seed words under the education theme include college entrance examination, textbooks, papers, etc., and the initial seed words under the life theme include food, transportation, clothing. Furthermore, before expanding the seed word list, you can first make an expansion judgment, for example Figure 3 A specific schematic diagram of expanding a seed word list is shown, which determines whether the initial seed word list meets the preset expansion condition. If it does, the initial seed word list is expanded using the target pre-trained model. If it does not meet the condition, the initial seed word list does not need to be expanded. In other words, the initial seed word list can be directly determined as the expanded seed word list. Among them, the preset expansion condition can be that the number of initial seed words in the initial seed word list is greater than the target threshold, and the target threshold is determined based on the required data volume. Specifically, a mapping relationship between the required data volume and the threshold can be established in advance. Specifically, the required data volume can be a first preset range of 20-50, and the corresponding threshold is 5. The required data volume is a second preset range of 51-100, and the corresponding threshold is 10. The target pre-trained model can specifically be a large language model.

[0077] Step S13: Based on the least to most hint algorithm, the part of speech of each target seed word in the expanded seed word list output by the thought chain algorithm is identified, and a hint information generation template corresponding to the part of speech of the target seed word is used to generate hint information corresponding to the target seed word, and the hint information is output by the thought chain algorithm.

[0078] In the present embodiment, prompt information corresponding to the target seed word is generated using a prompt information generation template corresponding to the part of speech of the target seed word, including: judging whether the part of speech of the target seed word belongs to a preset part of speech category; if the part of speech of the target seed word belongs to the preset part of speech category, then using a general prompt information generation template to generate general prompt information corresponding to the target seed word; if the part of speech of the target seed word does not belong to the preset part of speech category, then using an exclusive prompt information generation template to generate exclusive prompt information corresponding to the target seed word.

[0079] Determine whether the part of speech of the target seed word belongs to a preset part of speech category, which may be adjective, adverb, verb, or noun. If the part of speech of the target seed word belongs to the preset part of speech category, a general prompt information generation template (NormalPrompt) is used to generate general prompt information (normalPrompt) corresponding to the target seed word. If the part of speech of the target seed word does not belong to the preset part of speech category, an exclusive prompt information generation template (OwnPrompt) is used to generate exclusive prompt information (ownPrompt) corresponding to the target seed word. The general prompt information generation template and the exclusive prompt information generation template may be pre-set according to different subject areas, data volumes, application scenarios, and the like.

[0080] In this embodiment, the prompt information is outputted using the thought chain algorithm, including: the general prompt information and / or the exclusive prompt information is outputted using the thought chain algorithm. It can be understood that the output of the previous step is the expanded seed word list, and the input of this step is the expanded seed word list. Further, after the general prompt information is generated using the general prompt information generation template and the exclusive prompt information is generated using the exclusive prompt information generation template, the general prompt information and / or the exclusive prompt information is outputted using the thought chain algorithm, that is, the output of this step is the general prompt information and / or the exclusive prompt information.

[0081] Step S14: Based on the prompt information output by the thought chain algorithm and using a network search method, a target data set corresponding to the required data volume is obtained.

[0082] In this embodiment, based on the prompt information output by the thinking chain algorithm and using the network search method to obtain the target data set corresponding to the required data volume, it includes: performing a network search on the general prompt information output by the thinking chain algorithm to obtain the network search content; determining the paragraph level search content and the non-paragraph level search content from the network search content, and determining the paragraph level search content as the first data set; inputting the exclusive prompt information output by the thinking chain algorithm and / or the prompt information corresponding to the non-paragraph level search content into the target pre-training model, so as to obtain the second data set output by the target pre-training model; generating a target data set including the first data set and the second data set.

[0083] The output of the previous step is prompt information, so the input of this step is prompt information, specifically general prompt information and / or exclusive prompt information. Figure 4A specific target data set acquisition schematic diagram is shown, in which a network search is performed on the general prompt information, and a target pre-trained model is used to generate a data set for the exclusive prompt information, wherein, if the network search content obtained by the network search for the general prompt information includes non-paragraph level search content, the non-paragraph level search content must also be generated using the target pre-trained model. The paragraph level search content can be directly determined as a data set, that is, a network search is performed using the general prompt information to obtain the network search content; the paragraph level search content and the non-paragraph level search content are determined from the network search content, and the paragraph level search content is determined as the first data set, and the prompt information corresponding to the exclusive prompt information and / or the non-paragraph level search content is input into the target pre-trained model, so as to obtain the second data set output by the target pre-trained model, and generate a target data set including the first data set and the second data set. It can be seen that the intelligent agent can expand words into sentences, and sentences can be expanded into paragraphs, realizing the logic of point to line and line to surface.

[0084] In this embodiment, generating a target data set including a first data set and a second data set includes: filtering out a third data set corresponding to a target subject area from a historical data set stored in a database, and generating a target data set including the first data set, the second data set, and the third data set. Further, the previously generated historical data set can be saved to a database, and when data collection is required later, the historical data set stored in the database can also be viewed, and the third data set corresponding to the target subject area can be filtered out from the historical data set. In this way, the target data set can specifically include the first data set, the second data set, and the third data set.

[0085] In this embodiment, after generating a target data set including a first data set, a second data set, and a third data set, it also includes: using a target scoring model to perform a quality score on the target data set to filter out a target sub-data set with a quality score greater than a preset threshold, and storing the target sub-data set in a database. It should be noted that not all data sets can be stored in a database. Taking the target data set currently obtained as an example, the target data set can be scored for quality using a target scoring model, or it can be scored manually, with a quality score of 0-10. Further, a target sub-data set with a quality score greater than a preset threshold is filtered out, for example, a target sub-data set with a quality score of 6-10 is filtered out, or a user can filter out a suitable target sub-data set according to a specific quality score, so that the target sub-data set is persisted in the database. It can be seen that the detection method of providing a scoring model and the detection method of manual intervention are more humane.

[0086] In addition to completing long-term memory functions such as scoring and persisting data sets, the memory module in the intelligent agent can also complete the short-term memory function of saving all log records during the data collection process, including session records, network search content, prompt information, seed word lists, etc.

[0087] Furthermore, the pseudo code for data collection is as follows:

[0088] Begin

[0089] userInput: string;

[0090] domain,size:= parserUserInput(userInput);

[0091] seedList:=getSeedsByDomain(domain);

[0092] dataset:[string];

[0093] For seed in seedList:

[0094] word:=validate(seed);

[0095] if word is Nouns or word is Verb or word is Adjectives or word isAdverbs:

[0096] normalPrompt:=getNormalPrompt(seed);

[0097] content:=call WEBSEARCH;

[0098] sizeTotal = sizeTotal+len(content);

[0099] dataset.append(content);

[0100] ownPrompt:=getOwnPrompt(seed);

[0101] content:=call LLM;

[0102] sizeTotal = sizeTotal+len(content);

[0103] dataset.append(content);

[0104] else:

[0105] ownPrompt:=getOwnPrompt(seed);

[0106] content:=call LLM;

[0107] sizeTotal = sizeTotal+len(content);

[0108] dataset.append(content);

[0109] if sizeTotal <size:

[0110] call extendSeed();

[0111] else:

[0112] output dataset;

[0113] End.

[0114] It can be seen that the present invention is applied to an intelligent body, including: using a few-sample learning algorithm to determine the target subject area and the required data volume corresponding to the data collection needs, and using a thinking chain algorithm to output the target subject area; determining an initial seed word list corresponding to the target subject area output by the thinking chain algorithm from a seed word manager, and using a target pre-training model to expand the initial seed word list, and using the thinking chain algorithm to output the expanded seed word list; based on the least to most prompt algorithm, identifying the part of speech of each target seed word in the expanded seed word list output by the thinking chain algorithm, and using a prompt information generation template corresponding to the part of speech of the target seed word to generate prompt information corresponding to the target seed word, and using the thinking chain algorithm to output the prompt information; based on the prompt information output by the thinking chain algorithm and using a network retrieval method, obtaining a target data set corresponding to the required data volume.

[0115] The beneficial effects are as follows: the present application is applied to an intelligent agent, and a few-sample learning algorithm is used to determine the target subject area and the required amount of data for each data collection requirement, so as to correctly extract the user's data collection requirements, and expand the initial seed list under the target subject area to obtain an expanded list containing sufficient target seeds, that is, to improve the seed diversity and coverage. Subsequently, based on the minimum to maximum prompt algorithm, a prompt information generation template corresponding to the part of speech of the target seed is used to generate prompt information corresponding to the target seed. On the one hand, the part of speech of simple target seeds can be preferentially identified to maintain high efficiency and accuracy, and on the other hand, the prompt information is more comprehensive and rich, so that based on the prompt information, the prompt information can be collected from the target pre-training model, Data collection is completed in two aspects: network retrieval. Network retrieval is to obtain corresponding data from public data, and the target pre-training model generation method can generate new relevant data based on existing data. The two-aspect data collection can ensure both the quality and quantity of the data. Furthermore, in the process of controlling the data using the thinking chain algorithm, the reasoning result of each step will be used as the input of the next step. That is to say, the input in the list expansion step is the target subject area output by the previous step, the input in the prompt information generation step is the expanded seed word list output by the previous step, and the input of the data set acquisition step is the prompt information output by the previous step, thus forming a coherent thinking process.

[0116] See also Figure 5 The embodiment of the present invention discloses a specific data collection method. Compared with the previous embodiment, this embodiment further explains and optimizes the technical solution. Applied to an intelligent agent, it includes:

[0117] Step S21: Use a few-sample learning algorithm to determine the target subject area and the required data volume corresponding to the data collection requirements, and use a thought chain algorithm to output the target subject area.

[0118] For example Figure 6The figure shows a specific structure diagram of an intelligent agent. The intelligent agent is mainly divided into four main modules: planning module, tool module, execution module and memory module. Each module has its own unique functions and mutual cooperation mechanism to ensure that the system can complete various complex tasks efficiently and accurately. The planning module is responsible for task planning and strategy formulation, including three methods: Few Shot, Chain of Thought (thinking chain algorithm) and Least to Most (least to most prompt algorithm); the tool module provides various necessary tools and platform support to ensure that the system can efficiently complete various tasks, including part of speech recognition, network retrieval, pre-trained model and seed word manager; the execution module is responsible for executing various tasks and operations of the system, and through interaction with users, it realizes various complex dialogues and task processing; the memory module is responsible for managing and storing various memory information of the system to ensure that the system can maintain the coherence and consistency of information in complex tasks.

[0119] The intelligent agent execution module includes a distributed execution module and a multi-round dialogue module. The distributed execution module executes the process according to the task workflow formed by the planning module to ensure the correct execution of the data set collection task; the multi-round dialogue module uses a multi-round dialogue method to communicate with the target pre-trained model, which makes it easier for the pre-trained model to accurately understand the user's demands.

[0120] Step S22: Determine from the seed word manager an initial seed word list corresponding to the target subject area outputted using the thought chain algorithm, and output the initial seed word list using the thought chain algorithm.

[0121] In the process of controlling data using the thought chain algorithm, the reasoning result of each step will be used as the input of the next step. In other words, in this step, the thought chain algorithm is used to output the initial seed word list, thus forming a coherent thought process.

[0122] Step S23: using the target pre-training model and the expansion prompt information generation template to expand each initial seed word in the initial seed word list output by the thought chain algorithm to obtain each candidate seed word, and output each candidate seed word by using the thought chain algorithm; wherein the initial seed word is a preset representative word under the target subject area.

[0123] The Chain of Thought module helps the agent maintain logical consistency in complex tasks through a coherent chain of thinking. The reasoning results of each step will be used as the input of the next step to form a coherent thinking process. The seeds of the field are obtained according to the extracted field name. If the number of field seed words does not meet the requirements, the initial seed words in the initial seed word list are expanded.

[0124] The tool modules include a seed word management module to support the process of determining target subject areas, initial seed word lists, seed expansion, etc.

[0125] In the process of expanding each initial seed word in the initial seed word list by using the target pre-trained model and the expansion prompt information generation template, it is necessary to determine whether the initial seed word list meets the preset expansion conditions. If so, the initial seed word list is expanded by using the target pre-trained model.

[0126] Step S24: Filter the candidate seed words output by the thought chain algorithm based on the preset filtering conditions to obtain the filtered seed words, and aggregate the filtered seed words to obtain the aggregated seed words; construct an expanded seed word list including the initial seed words and the aggregated seed words, and output the expanded seed word list using the thought chain algorithm.

[0127] Based on preset filtering conditions, each candidate seed word is filtered to obtain each filtered seed word. Specifically, redundant candidate seed words that are not related to the topic are filtered, and each filtered seed word is aggregated to obtain each aggregated seed word, and then an expanded seed word list including the initial seed words and the aggregated seed words is constructed.

[0128] Step S25: Based on the least-to-most hint algorithm, the part of speech of each target seed word in the expanded seed word list output by the thought chain algorithm is identified, and the hint information corresponding to the target seed word is generated using a hint information generation template corresponding to the part of speech of the target seed word.

[0129] The Least to Most algorithm is an algorithm based on the least to most hints. It solves simple problems first and then gradually overcomes complex problems to ensure that the system remains efficient and accurate during the processing process. The Least to Most algorithm will sort the initial seed words and give priority to simple seed words. The collection process is consistent with Chain of Thought.

[0130] The tool module can identify the part of speech and generate prompt information. First, for the expanded seed word list, the part of speech of each target seed word is identified one by one, because the accurate judgment of the part of speech will directly affect the quality and applicability of the subsequent prompt information generation. After the part of speech identification is completed, the corresponding prompt information generation template should be matched according to the part of speech of the target seed word. These templates can be pre-designed. They specify how to generate prompt information that is both grammatical and easy for users to understand according to the characteristics of different parts of speech. For example, for noun target seed words, the template may emphasize the specific things or concepts they refer to, and for verb target seed words, the template may highlight their actions or state changes. Finally, using these prompt information generation templates corresponding to the part of speech, each target seed word is substituted into the template to generate corresponding prompt information. These prompt information are intended to better understand the meaning and usage of the target seed words and improve the efficiency and accuracy of information retrieval or text processing.

[0131] Step S26: Based on the prompt information output by the thought chain algorithm and using a network search method, a target data set corresponding to the required data volume is obtained.

[0132] The tool module can be used to generate data sets, responsible for expanding seed words into sentences and paragraphs, and realizing the logic of point-to-line and line-to-surface. Point-to-line: After the seed words are inserted into the corresponding prompt information generation template, they are used as the input parameters of the network search. After the network search returns, the results are parsed. The parsed results may be non-paragraph level (i.e. sentence level) or paragraph level. If it is paragraph level, the process ends. If it is sentence level, it continues to expand. Among them, network search supports two methods: search engine and bing. The detailed process is as follows:

[0133] 1) Determine the part of speech of a given word to facilitate the application of different prompt templates in subsequent processes;

[0134] 2) The prompt information generation template is divided into exclusive templates and general templates; the general template calls network retrieval for expansion, and the exclusive template calls the pre-trained model for expansion;

[0135] 3) After inserting the general template and the exclusive template according to the part of speech of the word, call the network search and pre-trained model to expand the sentence. If the result of the network search is already at the paragraph level, the workflow stops; if the result of the network search is at the sentence level, insert the general template and call the pre-trained model interface;

[0136] Furthermore, from line to surface: if the result obtained through network retrieval is at the sentence level, the prompt information generation template is inserted, and a dialogue is conducted with the target pre-trained model to continue to expand the sentence to the paragraph level. That is, if the result of network retrieval is at the sentence level, the prompt information generation template of this stage is inserted, and then the interface of the target pre-trained model is called to continue to expand it to the paragraph level.

[0137] For example Figure 7 A specific data collection schematic diagram is shown. It should be noted that if the generated target data set does not meet the required data volume, it is necessary to continue to obtain the target data set. The method of continued acquisition can be to further expand the expanded seed word list, and then continue to identify the part of speech and generate prompt information to obtain the target data set. In other words, it is determined whether the target data set is equal to the required data volume. If not, the expanded seed word list is determined as the new initial seed word list, and then the step of using the target pre-trained model and the expanded prompt information generation template to expand each initial seed word in the initial seed word list is jumped again.

[0138] It can be seen that the present application reasonably obtains the target data set, which can be used in the pre-training and fine-tuning stages; in the entire workflow of the intelligent agent, the present application uses open source models and normally usable network retrieval services, so compared with the previous method of obtaining data sets through crawlers, it is completely scientific and legal; the input of the intelligent agent is a natural language description of humans, which does not require additional standardized parameters and words, and is simple and convenient; by utilizing the general capabilities of the target pre-training model, the number of seeds and the content of the seeds can be expanded, thereby ensuring that high-quality and large-quantity data sets can be obtained.

[0139] Figure 8 A schematic diagram of a data collection device provided by an embodiment of the present invention, applied to an intelligent agent, includes:

[0140] The demand determination module 11 is used to determine the target subject area and the required data volume corresponding to the data collection demand by using a few-sample learning algorithm, and output the target subject area by using a thinking chain algorithm;

[0141] A seed word expansion module 12 is used to determine an initial seed word list corresponding to the target subject area output by the thought chain algorithm from the seed word manager, expand the initial seed word list by using the target pre-trained model, and output the expanded seed word list by using the thought chain algorithm;

[0142] A prompt generation module 13 is used to identify the part of speech of each target seed word in the expanded seed word list output by the thought chain algorithm based on the least to most prompt algorithm, and generate prompt information corresponding to the target seed word by using a prompt information generation template corresponding to the part of speech of the target seed word, and output the prompt information by using the thought chain algorithm;

[0143] The data set acquisition module 14 is used to acquire a target data set corresponding to the required data volume based on the prompt information output by the thought chain algorithm and by using a network search method.

[0144] It can be seen that the present invention is applied to an intelligent body, including: using a few-sample learning algorithm to determine the target subject area and the required data volume corresponding to the data collection needs, and using a thinking chain algorithm to output the target subject area; determining an initial seed word list corresponding to the target subject area output by the thinking chain algorithm from a seed word manager, and using a target pre-training model to expand the initial seed word list, and using the thinking chain algorithm to output the expanded seed word list; based on the least to most prompt algorithm, identifying the part of speech of each target seed word in the expanded seed word list output by the thinking chain algorithm, and using a prompt information generation template corresponding to the part of speech of the target seed word to generate prompt information corresponding to the target seed word, and using the thinking chain algorithm to output the prompt information; based on the prompt information output by the thinking chain algorithm and using a network retrieval method, obtaining a target data set corresponding to the required data volume.

[0145] The beneficial effects are as follows: the present application is applied to an intelligent agent, and a few-sample learning algorithm is used to determine the target subject area and the required amount of data for each data collection requirement, so as to correctly extract the user's data collection requirements, and expand the initial seed list under the target subject area to obtain an expanded list containing sufficient target seeds, that is, to improve the seed diversity and coverage. Subsequently, based on the minimum to maximum prompt algorithm, a prompt information generation template corresponding to the part of speech of the target seed is used to generate prompt information corresponding to the target seed. On the one hand, the part of speech of simple target seeds can be preferentially identified to maintain high efficiency and accuracy, and on the other hand, the prompt information is more comprehensive and rich, so that based on the prompt information, the prompt information can be collected from the target pre-training model, Data collection is completed in two aspects: network retrieval. Network retrieval is to obtain corresponding data from public data, and the target pre-training model generation method can generate new relevant data based on existing data. The two-aspect data collection can ensure both the quality and quantity of the data. Furthermore, in the process of controlling the data using the thinking chain algorithm, the reasoning result of each step will be used as the input of the next step. That is to say, the input in the list expansion step is the target subject area output by the previous step, the input in the prompt information generation step is the expanded seed word list output by the previous step, and the input of the data set acquisition step is the prompt information output by the previous step, thus forming a coherent thinking process.

[0146] Furthermore, the present application also discloses an electronic device. Fig. 9 This is a structural diagram of an electronic device according to an exemplary embodiment, and the content in the figure cannot be regarded as any limitation on the scope of use of this application. The electronic device may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. Among them, the memory 22 is used to store a computer program, and the computer program is loaded and executed by the processor 21 to implement the relevant steps in the data collection method disclosed in any of the aforementioned embodiments. In addition, the electronic device in this embodiment may specifically be an electronic computer.

[0147] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device; the communication interface 24 can create a data transmission channel between the electronic device and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0148] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0149] The operating system 221 is used to manage and control various hardware devices on the electronic device and the computer program 222, which can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to complete the data collection method performed by the electronic device disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks.

[0150] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the aforementioned disclosed data collection method. The specific steps of the method can refer to the corresponding contents disclosed in the aforementioned embodiments, and will not be repeated here.

[0151] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0152] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0153] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0154] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0155] The technical solution provided by the present application is introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technicians in this field, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A data collection method, characterized in that: Applied to intelligent agents, including: Use a few-shot learning algorithm to determine the target subject area and the amount of data required for data collection, and use a thought chain algorithm to output the target subject area; Determining an initial seed word list corresponding to the target subject area outputted by the thought chain algorithm from the seed word manager, expanding the initial seed word list by using the target pre-trained model, and outputting the expanded seed word list by using the thought chain algorithm; Based on the least-to-most prompt algorithm, the part of speech of each target seed word in the expanded seed word list output by the thought chain algorithm is identified, and prompt information corresponding to the target seed word is generated by using a prompt information generation template corresponding to the part of speech of the target seed word, and the prompt information is output by using the thought chain algorithm; Based on the prompt information output by the thought chain algorithm and using a network search method, a target data set corresponding to the required data volume is obtained.

2. The data collection method according to claim 1, characterized in that: The step of generating prompt information corresponding to the target seed word by using a prompt information generation template corresponding to the part of speech of the target seed word comprises: Determining whether the part of speech of the target seed word belongs to a preset part of speech category; If the part of speech of the target seed word belongs to the preset part of speech category, then using the general prompt information generation template to generate general prompt information corresponding to the target seed word; If the part of speech of the target seed word does not belong to the preset part of speech category, using the exclusive prompt information generation template to generate exclusive prompt information corresponding to the target seed word; Accordingly, the prompt information is outputted using a thought chain algorithm, including: The general prompt information and / or the exclusive prompt information are outputted using a thought chain algorithm.

3. The data collection method according to claim 2, characterized in that: The method of obtaining a target data set corresponding to the required data volume by using a network search method based on the prompt information output by the thought chain algorithm includes: Performing a network search on the general prompt information output by the thought chain algorithm to obtain network search content; Determine paragraph-level search content and non-paragraph-level search content from the network search content, and determine the paragraph-level search content as a first data set; Inputting the exclusive prompt information output by the thought chain algorithm and / or the prompt information corresponding to the non-paragraph level search content into the target pre-training model, so as to obtain a second data set output by the target pre-training model; A target data set including the first data set and the second data set is generated.

4. The data collection method according to claim 3, characterized in that: The generating a target data set including the first data set and the second data set comprises: Filtering a third data set corresponding to the target subject area from the historical data sets stored in the database, and generating a target data set including the first data set, the second data set and the third data set; Accordingly, after generating the target data set including the first data set, the second data set and the third data set, the method further includes: The target data set is quality scored using a target scoring model to screen out a target sub-data set whose quality score is greater than a preset threshold, and the target sub-data set is stored in the database.

5. The data collection method according to claim 1, characterized in that: The expanding the initial seed word list by using the target pre-trained model includes: Expanding each initial seed word in the initial seed word list by using the target pre-training model and the expansion prompt information generation template to obtain each candidate seed word; wherein the initial seed word is a preset representative word under the target subject area; Filtering each of the candidate seed words based on a preset filtering condition to obtain each filtered seed word, and aggregating each of the filtered seed words to obtain each aggregated seed word; An expanded seed word list including the initial seed words and the aggregated seed words is constructed.

6. The data collection method according to any one of claims 1 to 5, characterized in that: The method of using a few-sample learning algorithm to determine the target subject area and the required data volume corresponding to the data collection requirements includes: Receiving a current data collection instruction through a preset interface, and extracting data collection requirements in the current data collection instruction using a few-sample learning algorithm; A target subject area corresponding to the data collection requirement is screened out from various preset subject areas built into the seed word manager, and a required amount of data corresponding to the data collection requirement is determined.

7. The data collection method according to claim 1, characterized in that: The method of using a few-sample learning algorithm to determine the target subject area and the required data volume corresponding to the data collection requirements includes: Extracting a first data collection requirement in a current data collection instruction using a few-sample learning algorithm; If the first data collection requirement meets the preset extraction success condition, the first subject area and the first data volume in the first data collection requirement are determined as the target subject area and the required data volume respectively; If the first data collection requirement does not meet the preset extraction success condition, extracting the second data collection requirement in the current data collection instruction by using a regular expression; If the second data collection requirement satisfies the preset extraction success condition, the second subject area and the second data volume in the second data collection requirement are respectively determined as the target subject area and the required data volume; If the second data collection requirement does not meet the preset extraction success condition, the target pre-trained model is used to extract the third data collection requirement in the current data collection instruction, and the third subject area and the third data amount in the third data collection requirement are respectively determined as the target subject area and the required data amount.

8. A data collection device, characterized in that: Applied to intelligent agents, including: A demand determination module is used to determine the target subject area and the required data volume corresponding to the data collection demand using a few-sample learning algorithm, and output the target subject area using a thought chain algorithm; A seed word expansion module, used to determine an initial seed word list corresponding to the target subject area output by the thought chain algorithm from the seed word manager, expand the initial seed word list by using the target pre-trained model, and output the expanded seed word list by using the thought chain algorithm; a prompt generation module, for identifying the part of speech of each target seed word in the expanded seed word list output by the thought chain algorithm based on the least-to-most prompt algorithm, generating prompt information corresponding to the target seed word by using a prompt information generation template corresponding to the part of speech of the target seed word, and outputting the prompt information by using the thought chain algorithm; The data set acquisition module is used to acquire a target data set corresponding to the required data volume based on the prompt information output by the thought chain algorithm and by using a network search method.

9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to execute the computer program to implement the steps of the data collection method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data collection method according to any one of claims 1 to 7 are implemented.

Images