A data generation method, device and electronic equipment

Abstract

The application discloses a data generation method and device and electronic equipment. The data generation method comprises the following steps: performing pairing processing on data with the same intention in a first data set to obtain a first data set; performing weighting processing on core words with different intentions and core words with the same intention in the first data set to obtain a first word set; and performing data generation processing on the first data set and the first word set based on a target sentence rule to obtain a second data set, wherein the target sentence rule is the same as a sentence rule in the first data set, and the data quantity of the second data set is greater than that of the first data set.

Description

Technical Field

[0001] This application relates to the field of information technology, and in particular to a data generation method, apparatus and electronic device. Background Technology

[0002] Deep learning places high demands on data. In engineering, insufficient data is a major challenge for deep learning models. Manually generating data is time-consuming and labor-intensive. Therefore, developing an accurate and highly generalizable automatic data generation method is extremely important. Summary of the Invention

[0003] In view of this, embodiments of this application provide a data generation method, apparatus, and electronic device.

[0004] According to a first aspect of this application, embodiments of this application provide a data generation method, including:

[0005] Pair data with the same intent in the first dataset to obtain the first dataset;

[0006] The first word set is obtained by weighting the core words with different intentions and the core words with the same intention in the first dataset.

[0007] The first dataset and the first word set are processed to generate a second dataset based on the target statement rules. The target statement rules are the same as those in the first dataset. The second dataset has a larger data volume than the first dataset.

[0008] Optionally, data generation processing is performed on the first data set and the first word set based on the target statement rules, including:

[0009] Perform statistical analysis on each data entry in the first dataset to determine the target statement rules;

[0010] The original data generation model is optimized based on the target statement rules to obtain the target data generation model;

[0011] The first dataset and the first word set are input into the target data generation model to obtain the second dataset.

[0012] Optionally, data generation processing is performed on the first data set and the first word set based on the target statement rules, including:

[0013] The first data set and the first word set were obtained in batches.

[0014] Input the first dataset and the first word set from the same batch into the target data generation model to obtain the various subsets of the second dataset.

[0015] Optionally, the original data generation model is optimized based on the target statement rules to obtain the target data generation model, including:

[0016] Data with the same intent in the third dataset are paired to obtain a second dataset, wherein the third dataset is the dataset obtained from the first dataset in each batch;

[0017] The core words with different intentions and core words with the same intention in the third dataset are weighted to obtain the second word set;

[0018] The original data generation model is trained based on the second dataset, the second word set, and the target sentence rules to obtain the target data generation model. In the second dataset, one data item from each paired data pair is used as a training sample for the original data generation model, and the other data item is used as a label for the training sample.

[0019] Optionally, the loss during the training process of the target data generation model includes: language model loss, data pair generation loss, and generation pair similarity loss;

[0020] Among them, the language model loss is used to characterize the degree of difference between the target sentence generated based on the second word set and the target sentence rules and the fluent sentence;

[0021] The data-pair generation loss is used to characterize the degree of difference between the target sentence and the training samples;

[0022] The generated similarity loss is used to characterize the degree of difference between the target sentence and the label.

[0023] Optionally, data with the same intent in the third dataset are paired to obtain a second dataset, including:

[0024] Combine every two data points with the same intent in the third dataset to obtain the second dataset.

[0025] Optionally, the steps to obtain the core words for each intent include:

[0026] Obtain the third dataset; the third dataset includes multiple data entries corresponding to multiple intents;

[0027] Based on multiple data points corresponding to each intent in the third dataset, the core words corresponding to each intent are determined.

[0028] Optionally, based on multiple data points corresponding to each intent in the third dataset, the core words corresponding to each intent are determined, including:

[0029] Based on multiple data points corresponding to each intent in the third dataset, each repeated word in the multiple data points of each intent is identified to obtain the core word corresponding to each intent; among them, the number of times the repeated word appears in the multiple data points of the intent is greater than the preset number.

[0030] According to a second aspect of this application, embodiments of this application provide a data generation apparatus, including:

[0031] The matching module is used to match data with the same intent in the first dataset to obtain the first data set;

[0032] The weighted processing module is used to weight core words with different intentions and core words with the same intention in the first dataset to obtain the first word set;

[0033] The generation and processing module is used to generate data from the first dataset and the first word set based on the target statement rules to obtain the second dataset; wherein the target statement rules are the same as the statement rules in the first dataset, and the amount of data in the second dataset is greater than that in the first dataset.

[0034] According to a third aspect of this application, embodiments of this application provide an electronic device, including:

[0035] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to cause the at least one processor to perform a data generation method as described in the first aspect or any embodiment of the first aspect.

[0036] The data generation method, apparatus, and electronic device provided in this application embodiment obtain a first data set by pairing data with the same intent in a first dataset; weighting core words with different intents and core words with the same intent in the first dataset to obtain a first word set; and generating data from the first data set and the first word set based on target sentence rules to obtain a second dataset. The target sentence rules are the same as the sentence rules in the first dataset, and the second dataset has a larger data volume than the first dataset. Thus, each piece of data in the generated second dataset contains both core words with the same intent and core words with different intents, i.e., a mixture of core words with multiple intents, and conforms to the target sentence rules, forming a fluent sentence. Compared to simple similar sentences with the same intent, it has more meaningful information gain and is less prone to overfitting, thereby enabling the automatic generation of accurate and highly generalizable data corresponding to the first dataset.

[0037] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0038] Figure 1 This is a flowchart illustrating a data generation method according to an embodiment of this application;

[0039] Figure 2 This is a schematic diagram illustrating the training of the target data generation model in an embodiment of this application;

[0040] Figure 3 This is a schematic diagram of the structure of a data generation device according to an embodiment of this application;

[0041] Figure 4 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0043] This application provides a data generation method, such as... Figure 1 As shown, it includes:

[0044] S101, pair data with the same intent in the first dataset to obtain the first data set.

[0045] In this embodiment, the first dataset includes multiple data entries corresponding to multiple intents.

[0046] In practice, every two data entries with the same intent in the first dataset can be combined to obtain the second dataset. For example, the intent to apply for a work permit in the first dataset includes three data entries: A, B, and C. Data entry A is: "Can you help me apply for a work permit?" Data entry B is: "Can you help me apply for a work permit?" Data entry C is: "I want to apply for a work permit." Pairing A and B together yields one data pair; pairing B and C together yields another data pair; combining A and C yields yet another data pair. This process continues, pairing multiple data entries with different intents in the first dataset to obtain the first dataset.

[0047] S102, weight the core words with different intentions and the core words with the same intention in the first dataset to obtain the first word set.

[0048] In some implementations, the first dataset includes multiple core words corresponding to each intent.

[0049] In other implementations, the core words for each intent in the first dataset can be calculated based on the first dataset. Specifically, based on multiple data points corresponding to each intent in the first dataset, each repeated word in the multiple data points for each intent can be determined to obtain the core words corresponding to each intent; wherein, each repeated word appears more than a preset number of times in the multiple data points for each intent.

[0050] In this embodiment, after determining the core words of each intent in the first dataset, the core words with different intents and the core words with the same intent can be weighted to obtain the first word set.

[0051] In practice, for each data pair in the first data set, multiple core words can be selected according to preset weights from the core words of the intent corresponding to that data pair and the core words of other intents, thus obtaining the core words corresponding to that data pair. The set of core words corresponding to each data pair forms the first word set.

[0052] S103, Based on the target statement rules, perform data generation processing on the first data set and the first word set to obtain the second dataset; wherein, the target statement rules are the same as the statement rules in the first dataset, and the data volume of the second dataset is greater than that of the first dataset.

[0053] In this embodiment, the target sentence rule is a rule for forming fluent sentences. For example, the target sentence rule stipulates that the generated data must include a subject, a predicate, and an object, with the subject preceding the predicate, the predicate preceding the object, the predicate being a verb, and the subject and object being nouns, etc.

[0054] In some implementations, the target statement rules can be obtained by performing statistical analysis on each data entry in the first dataset.

[0055] In another implementation, the target statement rules can be obtained in advance, for example, by performing statistical analysis on each data entry in a third dataset.

[0056] In practice, the first dataset includes data pairs for each intent, and the first word set includes the core words corresponding to each data pair. Therefore, when generating data from the first dataset and the first word set based on target statement rules, for each data pair, generation processing can be performed on that data pair and its corresponding core words based on the target statement rules to obtain new data for that data pair. This new data has the same intent as the data pair, i.e., the same main theme; and the content of this new data has more meaningful information gain compared to the data content in the data pair. The collection of all the new data, or the collection of all the new data and the data in the first dataset, forms the second dataset.

[0057] The data generation method provided in this application embodiment obtains a first data set by pairing data with the same intent in a first dataset; a first word set by weighting core words with different intents and core words with the same intent in the first dataset; and a second dataset by generating data based on target sentence rules on the first data set and the first word set. The target sentence rules are the same as the sentence rules in the first dataset, and the second dataset has a larger data volume than the first dataset. Thus, each piece of data in the generated second dataset contains both core words with the same intent and core words with different intents, i.e., a mixture of core words with multiple intents, and conforms to the target sentence rules, forming a fluent sentence. Compared to simple similar sentences with the same intent, it has more meaningful information gain and is less prone to overfitting, thereby enabling the automatic generation of accurate and highly generalizable data corresponding to the first dataset.

[0058] In an optional embodiment, step S103, which involves performing data generation processing on the first data set and the first word set based on the target statement rules, includes:

[0059] S1031, Perform statistical analysis on each data entry in the first dataset to determine the target statement rules.

[0060] S1032, The original data generation model is optimized based on the target statement rules to obtain the target data generation model.

[0061] S1033, Input the first dataset and the first word set into the target data generation model to obtain the second dataset.

[0062] In this embodiment, step S1031 involves determining the target statement rule by performing statistical analysis on each data item in the first dataset when the first dataset is obtained.

[0063] Regarding step S1032, when optimizing the original data generation model based on the target sentence rules to obtain the target data generation model, the original data generation model can be trained based on the target sentence rules to obtain the target data generation model. Training the original data generation model based on the target sentence rules enables the trained target data generation model to generate fluent sentences that tend to conform to the target sentence rules.

[0064] Regarding step S1033, in specific implementation, each data pair in the first dataset and the corresponding core words of each data pair can be sequentially input into the target data generation model to generate new data corresponding to each data pair, thus obtaining the second dataset. For example, inputting data pair i "A, Can you help me apply for a work permit? B, Can you help me apply for a work permit?" and the corresponding core words "work permit, application, new employee" into the target data model can produce new data such as "I am a new employee, and I am still waiting for my work permit. Can you help me apply for a new work permit?"

[0065] In this embodiment, the original data generation model is optimized by target statement rules to obtain a target data generation model. Then, the target data generation model is used to generate a second dataset, thereby obtaining a general model that can automatically generate a new second dataset with the same intent based on the first data set and the first word set. This allows for the rapid and accurate generation of new data with the same intent.

[0066] In an optional embodiment, step S103, which involves performing data generation processing on the first data set and the first word set based on the target statement rules, includes:

[0067] The first dataset and the first word set are obtained in batches; the first dataset and the first word set in the same batch are input into the target data generation model to obtain the various subsets of the second dataset.

[0068] In this embodiment, pairing data with the same intent in the first dataset to obtain the first data set is done in batches. Furthermore, after obtaining each batch of the first data set, the core words with different intents and core words with the same intent in the first dataset are weighted according to that batch of the first data set to obtain the first word set corresponding to that batch of the first data set.

[0069] For example, the first dataset can be divided into three batches. The first batch includes multiple data entries for one intention, the second batch includes multiple data entries for another intention, and the third batch includes multiple data entries for the remaining intentions. Alternatively, the first batch may include a portion of the data entries for each intention, the second batch may include another portion of the data entries for each intention, and the third batch may include the remaining data entries for each intention.

[0070] First, data with the same intent in the first batch of data are paired to obtain the first dataset. Then, for each data pair in the first dataset, multiple core words are selected according to preset weights from the core words of the intent corresponding to that data pair and the core words of other intents, thus obtaining the core words corresponding to that data pair. The set of core words corresponding to each data pair forms the first word set of the first batch. Then, the first dataset and the first word set are input into the target data generation model to obtain the first subset of the second dataset. This process is repeated for the second and third batches of data to obtain the second and third subsets of the second dataset. The set of the first, second, and third subsets forms the second dataset.

[0071] In this embodiment, the first data set and the first word set are obtained in batches, so that the amount of data in each batch of the first data set and the first word set is not too large, which makes it easier for the target data generation model to process.

[0072] In an optional embodiment, step S1032, optimizing the original data generation model based on the target statement rules to obtain the target data generation model, includes:

[0073] Data with the same intent in the third dataset are paired to obtain the second dataset. The third dataset is the dataset obtained from the first dataset in each batch. Core words with different intents and core words with the same intent in the third dataset are weighted to obtain the second word set. The original data generation model is trained based on the second dataset, the second word set, and the target sentence rules to obtain the target data generation model. In the second dataset, one data pair in the paired data pair is used as the training sample of the original data generation model, and the other data pair is used as the label of the training sample.

[0074] In practice, a third dataset can be obtained first. The third dataset can be a set of data that is selectively or randomly extracted from the first dataset in batches, with each batch yielding a third dataset. The third dataset includes multiple data points corresponding to multiple intents. Then, based on the multiple data points for each intent in the third dataset, the data pairs for each intent are determined. For example, one data pair i for applying for a work card is: A, Can you help me apply for a work card? B, Can you help me apply for a work card?

[0075] In some implementations, pairing data with the same intent in the third dataset to obtain a second dataset includes: combining every two data pairs with the same intent in the third dataset to obtain the second dataset.

[0076] In this embodiment, each pair of data with the same intent is combined, so that each intent can obtain more data pairs, resulting in more data pairs in the second data set, and thus generating more new data.

[0077] Then, for each data pair of each intent, multiple core words are selected according to preset weights from the core words of that intent and the core words of other intents, forming a second word set. For example, for data pair i, the core words "work card, application..." are selected from the core words "work card, application..." with a weight of 2 / 3, and the core word "new employee" is selected from the core words "employee, new employee, processing, HR..." with a weight of 1 / 3, resulting in the core words "work card, application, new employee" for data pair i.

[0078] In some implementations, the step of obtaining the core words for each intent may include:

[0079] Obtain a third dataset; the third dataset includes multiple data points corresponding to multiple intents; based on the multiple data points corresponding to each intent in the third dataset, determine the core words corresponding to each intent.

[0080] In this implementation, the core words corresponding to each intent are determined based on multiple data points in the third dataset corresponding to each intent. Specifically, this includes: determining each repeated word in the multiple data points of each intent based on the multiple data points of each intent in the third dataset, thereby obtaining the core words corresponding to each intent; wherein the repeated word appears more than a preset number of times in the multiple data points of the intent.

[0081] For example, the intention to apply for a work permit includes three data points: A, B, and C. Data point A is: "Can you help me apply for a work permit?" Data point B is: "Can you help me apply for a work permit?" Data point C is: "I want to apply for a work permit." Through statistical analysis of multiple data points related to the intention to apply for a work permit, we can identify the recurring words "apply" and "work permit." Furthermore, these two recurring words appear more than twice in all three data points. Therefore, the core words corresponding to the intention to apply for a work permit are "apply" and "work permit."

[0082] In this implementation, since core words frequently appear in data corresponding to the same intent, the core words corresponding to each intent can be obtained quickly and accurately by extracting repeated words from multiple data points corresponding to each intent in the third dataset.

[0083] Then, based on each data pair for each intent, and the corresponding core words and target sentence rules, the original data generation model can be trained to obtain the target data generation model, such as... Figure 2 As shown in the diagram. In this data pair, one data point serves as the training sample for the original data generation model, and the other data point serves as the label for the training sample.

[0084] In some implementations, the second data set and the second word set can be obtained in batches, and the original data generation model can be trained based on the second data set, the second word set and the target sentence rules in the same batch. By training the second data set, the second word set and the target sentence rules in multiple batches, the final target data generation model can be obtained.

[0085] In this embodiment, by training the original data generation model using a second data set, a second word set, and target sentence rules, a target data generation model can be obtained. This target data generation model can automatically generate a second dataset based on the first data set and the corresponding first word set.

[0086] In an optional embodiment, the losses during the training process of the target data generation model include: language model loss, data pair generation loss, and generation pair similarity loss;

[0087] Among them, the language model loss is used to characterize the degree of difference between the target sentence generated based on the second word set and the target sentence rules and the fluent sentence; the data pair generation loss is used to characterize the degree of difference between the target sentence and the training samples; and the generation pair similarity loss is used to characterize the degree of difference between the target sentence and the label.

[0088] In this embodiment, during the training process of the target data generation model, by calculating the language model loss, the target data generation model can generate fluent new data that tends to follow the target syntax rules; by calculating the data pair generation loss, the target data generation model can generate new data with the same intent, that is, ensure that the core words of the intent remain unchanged; by calculating the generation pair similarity loss, the incorporated perturbations, that is, the core words of different intents, can be ensured that they do not destroy the main idea of ​​the sentence.

[0089] This application also provides a data generation apparatus, such as... Figure 3 As shown, it includes:

[0090] The matching module 31 is used to match data with the same intent in the first dataset to obtain the first data set;

[0091] The weighted processing module 32 is used to perform weighted processing on core words with different intentions and core words with the same intention in the first dataset to obtain the first word set;

[0092] The generation processing module 33 is used to perform data generation processing on the first data set and the first word set based on the target statement rules to obtain the second dataset; wherein the target statement rules are the same as the statement rules in the first dataset, and the data volume of the second dataset is larger than that of the first dataset.

[0093] The data generation apparatus provided in this application provides a first data set by pairing data with the same intent in a first dataset; a first word set by weighting core words with different intents and core words with the same intent in the first dataset; and a second dataset by generating data based on target sentence rules on the first data set and the first word set. The target sentence rules are the same as the sentence rules in the first dataset, and the second dataset has a larger data volume than the first dataset. Thus, each piece of data in the generated second dataset contains both core words with the same intent and core words with different intents, i.e., a mixture of core words with multiple intents, and conforms to the target sentence rules, forming a fluent sentence. Compared to simple similar sentences with the same intent, it has more meaningful information gain and is less prone to overfitting, thereby enabling the automatic generation of accurate and highly generalizable data corresponding to the first dataset.

[0094] According to embodiments of this application, this application also provides an electronic device and a readable storage medium.

[0095] Figure 4A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of this application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0096] like Figure 4 As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. The RAM 803 may also store various programs and data required for the operation of the electronic device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0097] Multiple components in electronic device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of displays, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows electronic device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0098] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as data generation methods. For example, in some embodiments, the data generation method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the data generation method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the data generation method by any other suitable means (e.g., by means of firmware).

[0099] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0100] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0101] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0102] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0103] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0104] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0105] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application can be achieved, and this is not limited herein.

[0106] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0107] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data generation method, comprising: Pair data with the same intent in the first dataset to obtain the first dataset; The first set of words is obtained by weighting the core words with different intentions and the core words with the same intention in the first dataset. Based on the target statement rules, the first dataset and the first word set are processed to generate a second dataset. The target statement rules are the same as those in the first dataset, and the second dataset has a larger data volume than the first dataset.

2. The method according to claim 1, wherein the data generation process based on the target statement rules for the first data set and the first word set includes: Perform statistical analysis on each data entry in the first dataset to determine the target statement rule; The original data generation model is optimized based on the target statement rules to obtain the target data generation model; The first dataset and the first word set are input into the target data generation model to obtain the second dataset.

3. The method according to claim 2, wherein the data generation process based on the target statement rule for the first data set and the first word set includes: The first data set and the first word set are obtained in batches; The first dataset and the first word set from the same batch are input into the target data generation model to obtain various subsets of the second dataset.

4. The method according to claim 3, wherein the original data generation model is optimized based on the target statement rule to obtain the target data generation model, comprising: Data with the same intent in the third dataset are paired to obtain a second dataset, wherein the third dataset is the dataset obtained from the first dataset in each batch; The core words with different intentions and the core words with the same intention in the third dataset are weighted to obtain the second word set; The original data generation model is trained based on the second data set, the second word set, and the target sentence rules to obtain the target data generation model; wherein, one data item in the paired data pair in the second data set is used as the training sample of the original data generation model, and the other data item is used as the label of the training sample.

5. The method according to claim 4, wherein the loss during the training process of the target data generation model includes: Language model loss, data pair generation loss, and generation pair similarity loss; The language model loss is used to characterize the degree of difference between the target sentence generated based on the second word set and the target sentence rule and a fluent sentence; The data pair generation loss is used to characterize the degree of difference between the target sentence and the training samples; The generated pair similarity loss is used to characterize the degree of difference between the target sentence and the label.

6. The method according to claim 4, wherein pairing data with the same intent in the third dataset to obtain a second dataset includes: The second data set is obtained by combining every two data points with the same intent in the third dataset.

7. The method according to claim 4, wherein the step of obtaining the core word of each intent includes: Obtain the third dataset; The third dataset includes multiple data entries corresponding to multiple intents; Based on multiple data points corresponding to each intent in the third dataset, the core words corresponding to each intent are determined.

8. The method according to claim 7, wherein the core words corresponding to each intent are determined based on multiple data points in the third dataset corresponding to each intent, including: Based on multiple data points corresponding to each intent in the third dataset, each repeated word in the multiple data points of each intent is determined to obtain the core word corresponding to each intent; wherein, the repeated word appears more than a preset number of times in the multiple data points of the intent.

9. A data generation apparatus, comprising: The matching module is used to match data with the same intent in the first dataset to obtain the first data set; The weighted processing module is used to perform weighted processing on core words with different intentions and core words with the same intention in the first dataset to obtain the first word set; The generation processing module is used to perform data generation processing on the first dataset and the first word set based on the target statement rules to obtain a second dataset; wherein the target statement rules are the same as the statement rules in the first dataset, and the data volume of the second dataset is greater than that of the first dataset.

10. An electronic device, comprising: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to cause the at least one processor to perform the data generation method as described in any one of claims 1-8.

Images