Text data extension method and device
By generating extended text data through methods that obtain extended constraints and transform semantic trees, the problem of insufficient consistency and diversity of extended text data in existing technologies is solved, achieving high-quality text data extension and improving model performance.
Patent Information
- Application Number
- CN202410931109.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-11
- Publication Date
- 2026-01-13
AI Technical Summary
Existing text data expansion methods struggle to simultaneously ensure the consistency and diversity of expanded text data, resulting in low quality of expanded text data.
By obtaining extended constraints from the original text data, constructing and transforming the original semantic tree, and using a pre-trained text data extension model to generate extended text data, combined with a fidelity threshold for filtering, the consistency and diversity of the generated extended text data are ensured.
It improves the quality of extended text data, maintains semantic consistency between generated and original text data, increases the diversity of text data, and enhances the model's generalization ability and robustness.
Smart Images
Figure CN121328467A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and in particular to a method and apparatus for expanding text data. Background Technology
[0002] Text data augmentation refers to the process of expanding original text data using specific methods and techniques to increase its diversity. Augmenting text data can improve the generalization ability of models, reduce the risk of overfitting, and enhance model performance in various natural language processing tasks. Therefore, text data augmentation is of great significance in natural language processing and text processing tasks. Summary of the Invention
[0003] In view of this, embodiments of this application provide a method and apparatus for expanding text data to improve the quality of expanded text data.
[0004] To achieve the above objectives, the technical solutions provided in this application are as follows:
[0005] In a first aspect, embodiments of this application provide a method for expanding text data, including:
[0006] Obtain the extended constraints corresponding to the original text data;
[0007] Construct the original semantic tree corresponding to the original text data;
[0008] The original semantic tree is transformed to obtain a transformed semantic tree;
[0009] The text data corresponding to the transformed semantic tree is expanded based on the extended constraints to generate extended text data corresponding to the original text data.
[0010] As an optional implementation of this application, the step of obtaining the extended constraint conditions corresponding to the original text data includes:
[0011] At least one attribute is determined based on a pre-defined diversity system;
[0012] Obtain the attribute information of at least one attribute corresponding to the original text data;
[0013] Based on the attribute information of at least one attribute, obtain a set of diverse elements of the original text data;
[0014] Based on the set of diverse elements, the extended constraints corresponding to the original text data are generated.
[0015] As an optional implementation of this application, the step of generating extended constraint conditions corresponding to the original text data based on the set of diverse elements includes:
[0016] At least one diversity element is randomly selected from the set of diversity elements;
[0017] Based on the at least one diversity element, the extended constraints corresponding to the original text data are generated.
[0018] As an optional implementation of this application, the step of obtaining the extended constraint conditions corresponding to the original text data includes:
[0019] The original text data and at least one sample text data are compared to obtain the comparison result between the original text data and at least one sample text data.
[0020] Based on the comparison results between the original text data and at least one sample text data, extended constraints corresponding to the original text data are generated.
[0021] As an optional implementation of this application, the transformation of the original semantic tree to obtain a transformed semantic tree includes:
[0022] The transformed semantic tree is obtained by randomly pruning one branch of the original semantic tree and / or swapping the positions of two elements under the same branch of the original semantic tree.
[0023] As an optional implementation of this application, the step of expanding the text data corresponding to the transformed semantic tree based on the extended constraints to generate extended text data corresponding to the original text data includes:
[0024] The extended constraints and the text data corresponding to the transformed semantic tree are input into a pre-trained text data extension model, and the extended text data corresponding to the original text data output by the text data extension model is obtained.
[0025] As an optional implementation of this application, after expanding the text data corresponding to the transformed semantic tree based on the extended constraints to generate extended text data corresponding to the original text data, the method further includes:
[0026] Obtain the degree of restoration of the extended text data relative to the original text data;
[0027] Determine whether the degree of restoration is greater than the degree of restoration threshold;
[0028] If not, then discard the extended text data.
[0029] As an optional implementation of this application, obtaining the degree of restoration of the extended text data relative to the original text data includes:
[0030] Obtain a first element set and a second element set, wherein the first element set is a set of 1 to n word groups of the original text data, and the second element set is a set of 1 to n word groups of the extended text data, where n is an integer greater than 1;
[0031] Calculate the intersection of the first set of elements and the second set of elements to obtain the third set of elements;
[0032] Calculate the ratio of the number of elements in the third element set to the number of elements in the first element set to obtain the coverage.
[0033] Calculate the ratio of the number of elements in the third element set to the number of elements in the second element set to obtain the similarity.
[0034] Calculate the product of the coverage and the similarity to obtain the degree of restoration of the extended text data relative to the original text data.
[0035] As an optional implementation of this application, obtaining the degree of restoration of the extended text data relative to the original text data includes:
[0036] The original text data and the extended text data are respectively input into the target sentence encoding model to obtain the original sentence vector corresponding to the original text data and the extended sentence vector corresponding to the extended text data output by the target sentence encoding model;
[0037] Calculate the cosine values of the original sentence vector and the extended sentence vector to obtain the degree of restoration of the extended text data relative to the original text data.
[0038] Secondly, embodiments of this application provide a text data expansion device, including:
[0039] The acquisition unit is used to acquire the extended constraints corresponding to the original text data.
[0040] The construction unit is used to construct the original semantic tree corresponding to the original text data;
[0041] A transformation unit is used to transform the original semantic tree to obtain a transformed semantic tree;
[0042] The generation unit is used to extend the text data corresponding to the transformed semantic tree based on the extended constraints to generate extended text data corresponding to the original text data.
[0043] As an optional implementation of this application, the acquisition unit is specifically used to determine at least one attribute based on a preset diversity system; acquire attribute information of the at least one attribute corresponding to the original text data; acquire a set of diversity elements of the original text data based on the attribute information of the at least one attribute; and generate extended constraint conditions corresponding to the original text data based on the set of diversity elements.
[0044] As an optional implementation of this application, the acquisition unit is specifically used to randomly select at least one diversity element from the set of diversity elements; and generate extended constraint conditions corresponding to the original text data based on the at least one diversity element.
[0045] As an optional implementation of this application, the acquisition unit is specifically used to compare the original text data and at least one sample text data to obtain the comparison result of the original text data and at least one sample text data; and to generate extended constraint conditions corresponding to the original text data based on the comparison result of the original text data and at least one sample text data.
[0046] As an optional implementation of this application, the transformation unit is specifically used to randomly prune a branch of the original semantic tree and / or swap the positions of two elements under the same branch of the original semantic tree to obtain the transformed semantic tree.
[0047] As an optional implementation of this application, the generation unit is specifically used to input the extended constraints and the text data corresponding to the transformed semantic tree into a pre-trained text data extension model, and to obtain the extended text data corresponding to the initial text data output by the text data extension model.
[0048] As an optional implementation of this application, the generation unit is further configured to, after expanding the text data corresponding to the transformed semantic tree based on the extended constraint conditions to generate extended text data corresponding to the original text data, obtain the degree of restoration of the extended text data relative to the original text data; determine whether the degree of restoration is greater than the degree of restoration threshold; if not, discard the extended text data.
[0049] As an optional implementation of this application, the generation unit is specifically used to obtain a first element set and a second element set, wherein the first element set is a set of 1 to n word groups of the original text data, and the second element set is a set of 1 to n word groups of the extended text data, where n is an integer greater than 1; calculate the intersection of the first element set and the second element set to obtain a third element set; calculate the ratio of the number of elements in the third element set to the number of elements in the first element set to obtain coverage; calculate the ratio of the number of elements in the third element set to the number of elements in the second element set to obtain similarity; and calculate the product of the coverage and the similarity to obtain the restoration degree of the extended text data relative to the original text data.
[0050] As an optional implementation of this application, the generation unit is specifically used to input the original text data and the extended text data into the target sentence encoding model respectively, obtain the original sentence vector corresponding to the original text data and the extended sentence vector corresponding to the extended text data output by the target sentence encoding model; calculate the cosine value of the original sentence vector and the extended sentence vector to obtain the degree of restoration of the extended text data relative to the original text data.
[0051] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor, wherein the memory is used to store a computer program and the processor is used to cause the electronic device to implement the text data expansion method described in any of the above embodiments when executing the computer program.
[0052] Fourthly, embodiments of this application provide a computer-readable storage medium that, when executed by a computing device, causes the computing device to implement the text data expansion method described in any of the above embodiments.
[0053] Fifthly, embodiments of this application provide a computer program product that, when run on a computer, enables the computer to implement the text data expansion method described in any of the above embodiments.
[0054] The text data expansion method provided in this application first obtains the expansion constraints corresponding to the original text data, then constructs the original semantic tree corresponding to the original text data, transforms the original semantic tree to obtain a transformed semantic tree, and then expands the text data corresponding to the transformed semantic tree based on the expansion constraints to generate expanded text data corresponding to the original text data. Because the text data expansion method provided in this application can obtain the expansion constraints corresponding to the original text data, transform the original semantic tree to obtain a transformed semantic tree, and expand the text data corresponding to the transformed semantic tree based on the expansion constraints, and because the expansion constraints can improve the consistency of the generated expanded text data, and the text data corresponding to the transformed semantic tree obtained by transforming the original semantic tree can improve the diversity of the generated expanded text data, this application embodiment can simultaneously ensure the consistency and diversity of the expanded text data, thereby improving the quality of the expanded text data. Attached Figure Description
[0055] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0056] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings that need to be called in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0057] Figure 1 One of the flowcharts for the text data expansion method provided in the embodiments of this application;
[0058] Figure 2 A second flowchart illustrating the text data expansion method provided in this application embodiment;
[0059] Figure 3 A schematic diagram of the structure of the text data expansion device provided in the embodiments of this application;
[0060] Figure 4 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation
[0061] To better understand the above-mentioned objectives, features, and advantages of this application, the solution of this application will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.
[0062] Many specific details are set forth in the following description in order to provide a full understanding of this application, but this application may also be implemented in other ways different from those described herein. Obviously, the embodiments in the specification are only some embodiments of this application, and not all embodiments.
[0063] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner. Furthermore, in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.
[0064] To improve model training performance and evaluation accuracy, text data expansion tasks need to ensure consistency between the expanded and original text data while increasing the diversity of the text data. However, commonly used text data expansion schemes such as synonym replacement, word insertion, and sentence restructuring often lack a deep understanding of the semantics and context of the original text data and are often based on rules. Therefore, it is often difficult to simultaneously guarantee the consistency and diversity of the expanded text data, resulting in a low level of quality for the expanded text data.
[0065] This application provides a method for expanding text data, referring to... Figure 1 As shown, the method for expanding this text data includes the following steps:
[0066] S11. Obtain the extended constraints corresponding to the original text data.
[0067] In some embodiments, obtaining the extended constraints corresponding to the original text data may include the following steps a to d:
[0068] Step a: Determine at least one attribute based on a pre-defined diversity system.
[0069] In some embodiments, the diversity system may include attributes such as word count, language style, sentence type, theme, entity, and domain.
[0070] Step b: Obtain the attribute information of at least one attribute corresponding to the original text data.
[0071] In some embodiments, the method for obtaining attribute information of at least one attribute corresponding to the original text data may include:
[0072] Construct a model prompt corresponding to the at least one attribute, and obtain attribute information of the at least one attribute corresponding to the original text data based on the original text data, the model prompt corresponding to the at least one attribute, and the pre-trained text processing model.
[0073] In some embodiments, the pre-trained text processing model can be a Large Language Model (LLM).
[0074] For example, when the original text data is "What is the capital of country C?", and the at least one attribute includes: word count, language style, sentence type, topic, entity, and domain, the attribute information of the at least one attribute corresponding to the original text data may include the following information:
[0075] Word count: This sentence consists of 9 characters;
[0076] Language style: This sentence uses a concise, direct, declarative language style;
[0077] Sentence type: This sentence is an interrogative sentence, ending with a question mark, used to inquire about information about the capital of country C;
[0078] Subject: The subject of this sentence is an inquiry about the capital of country C;
[0079] Entities: The entities involved in this sentence are "Country C" and "Capital". "Country C" is a country entity, and "Capital" is a geographical concept entity.
[0080] Field: The field this sentence refers to is geography.
[0081] Step c: Obtain the set of diverse elements of the original text data based on the attribute information of the at least one attribute.
[0082] As described in the example above, the set of diverse elements of the original text data obtained based on the attribute information of at least one attribute can be {consisting of 9 characters; concise and direct declarative language style; interrogative sentence; inquiry about the capital of country C; the entities involved are "country C" and "capital"; geographical domain}.
[0083] Step d: Generate extended constraints corresponding to the original text data based on the set of diverse elements.
[0084] In some embodiments, generating extended constraints corresponding to the original text data based on the set of diverse elements includes: randomly selecting at least one diverse element from the set of diverse elements; and generating extended constraints corresponding to the original text data based on the at least one diverse element.
[0085] For example, if the diversity elements randomly selected from the set of diversity elements include "composed of 9 characters", "interrogative sentence", and "geographical domain", then the generated extended constraints can be: the number of characters is in the range of [9-2, 9+2], the sentence is an interrogative sentence, and it involves a geographical domain.
[0086] The above embodiments randomly select at least one diversity element from the set of diversity elements, and generate extended constraints corresponding to the original text data based on the at least one diversity element. Different combinations of diversity elements can generate different extended constraints, thereby improving the diversity of the extended text data.
[0087] In some embodiments, obtaining the extended constraints corresponding to the original text data may include the following steps 1 to 2:
[0088] Step 1: Compare the original text data with at least one sample text data to obtain the comparison result of the original text data and at least one sample text data.
[0089] In some embodiments, the original text data and at least one sample text data can be compared using a pre-trained text processing model to obtain a comparison result between the original text data and at least one sample text data.
[0090] For example, if the original text data is "What is the capital of country C?", and the at least one sample text data includes "What is the largest city in province S?" and "In which district is the tallest building in city B?", then the comparison results of sentence 1 "What is the capital of country C?", sentence 2 "What is the largest city in province S?", and sentence 3 "In which district is the tallest building in city B?" can be as follows:
[0091] Differences:
[0092] a. Different themes: Sentence 1 asks about the capital of country C, sentence 2 asks about the largest city in province S, and sentence 3 asks about the district where the tallest building in city B is located.
[0093] b. Different entities: Sentence 1 involves the entities of country C and its capital, sentence 2 involves the entities of province S and city S, and sentence 3 involves the entities of city B and the district where the building is located.
[0094] c. The objects of inquiry are different: Sentence 1 inquires about the capital of country C, sentence 2 inquires about the largest city in province S, and sentence 3 inquires about the district where the tallest building in city B is located.
[0095] Similarities:
[0096] a. Same sentence structure: All three sentences are interrogative sentences, used to inquire about specific information.
[0097] b. Similar grammatical structure: All three sentences use a similar grammatical structure, namely, "What is Y of X?", where X represents a geographical region and Y represents a specific attribute or information.
[0098] c. Same information needs: All three sentences express a need for specific information, namely the capital of country C, the largest city in province S, and the district where the tallest building in city B is located.
[0099] Step 2: Based on the comparison results between the original text data and at least one sample text data, generate extended constraint conditions corresponding to the original text data.
[0100] In some embodiments, generating extended constraints corresponding to the original text data based on the comparison results of the original text data and at least one sample text data includes: processing the comparison results of the original text data and at least one sample text data based on a preset rule script to obtain the extended constraints corresponding to the original text data.
[0101] As described in the above embodiments, the extended constraints corresponding to the original text data generated based on the comparison results of the original text data and at least one sample text data may include: the sentence structure is an interrogative sentence, involving "country C" and "capital", and expressing a need for specific information.
[0102] S12. Construct the original semantic tree corresponding to the original text data.
[0103] In some embodiments, constructing the original semantic tree corresponding to the original text data includes: obtaining the original semantic tree corresponding to the original text data through a semantic tree extraction algorithm.
[0104] For example, the semantic tree extraction algorithm can be the Natural Language Toolkit (NLTK).
[0105] For example, if the original text data is "The sun has just risen over the mountaintop, shrouded in the bright red morning glow," then the original semantic tree corresponding to the original text data can be constructed as follows:
[0106]
[0107]
[0108] S13. Transform the original semantic tree to obtain a transformed semantic tree.
[0109] In some embodiments, transforming the original semantic tree includes randomly pruning a branch of the original semantic tree.
[0110] As in the example above, by pruning the branches in rows 19 and 20 of the original semantic tree, the resulting transformed semantic tree contains the text "The sun has just risen over the mountain and is shrouded in the morning glow"; for another example, by pruning the branches in rows 6 and 7 of the original semantic tree, the resulting transformed semantic tree contains the text "The sun has risen over the mountain and is shrouded in the bright red morning glow".
[0111] In some embodiments, transforming the original semantic tree includes swapping the positions of two elements on the same branch of the original semantic tree.
[0112] As described in the previous example, by swapping the positions of the elements in rows 6-13 and rows 14-24 of the original semantic tree, the resulting transformed semantic tree corresponds to the text content "The sun was veiled by the bright red morning glow, just rising over the mountaintop".
[0113] In some embodiments, transforming the original semantic tree includes: randomly pruning a branch of the original semantic tree and swapping the positions of two elements under the same branch of the original semantic tree.
[0114] Following the example above, by pruning the branches in rows 19 and 20 of the original semantic tree and swapping the positions of the elements in rows 6 to 13 with the elements in rows 14 to 24, the resulting transformed semantic tree corresponds to the text content "The sun was veiled by the morning glow, just rising over the mountaintop".
[0115] S14. Based on the extended constraints, the text data corresponding to the transformed semantic tree is extended to generate extended text data corresponding to the original text data.
[0116] In some embodiments, the text data corresponding to the transformed semantic tree is expanded based on the extended constraints to generate extended text data corresponding to the original text data, including:
[0117] The extended constraints and the text data corresponding to the transformed semantic tree are input into a pre-trained text data extension model, and the extended text data corresponding to the original text data output by the text data extension model is obtained.
[0118] In some embodiments, the pre-trained text data expansion model can be a large language model.
[0119] For example, if the text data corresponding to the transformed semantic tree is "The sun rises over the mountaintop, veiled by the morning glow," and the extended constraints include: less than 30 characters, colloquial, relating to morning, and relating to scenery, then the extended text data corresponding to the initial text data may include:
[0120] 1. The sun rises over the mountaintop, and the morning glow is as beautiful as a painting.
[0121] 2. As the sun rises over the mountain in the morning, the rosy clouds illuminate the breathtaking scenery.
[0122] 3. The morning sun rises, casting a beautiful glow on the sky.
[0123] The text data expansion method provided in this application first obtains the expansion constraints corresponding to the original text data, then constructs the original semantic tree corresponding to the original text data, transforms the original semantic tree to obtain a transformed semantic tree, and then expands the text data corresponding to the transformed semantic tree based on the expansion constraints to generate expanded text data corresponding to the original text data. Because the text data expansion method provided in this application can obtain the expansion constraints corresponding to the original text data, transform the original semantic tree to obtain a transformed semantic tree, and expand the text data corresponding to the transformed semantic tree based on the expansion constraints, and because the expansion constraints can improve the consistency of the generated expanded text data, and the text data corresponding to the transformed semantic tree obtained by transforming the original semantic tree can improve the diversity of the generated expanded text data, this application embodiment can simultaneously ensure the consistency and diversity of the expanded text data, thereby improving the quality of the expanded text data.
[0124] In this application's embodiments, diversity refers to the degree of difference between the generated extended text data. In text data extension tasks, diversity means that the generated extended text data have different semantics, structures, or expressions. Increased diversity can help the model better learn data variations and generalize, thereby improving the model's performance and robustness.
[0125] As an extension and refinement of the above embodiments, this application provides another method for expanding text data, referring to... Figure 2 As shown, the method for expanding this text data includes the following steps:
[0126] S201. Determine at least one attribute based on a pre-defined diversity system.
[0127] S202. Obtain the attribute information of at least one attribute corresponding to the original text data.
[0128] S203. Obtain the set of diverse elements of the original text data based on the attribute information of the at least one attribute.
[0129] S204. Randomly select at least one diversity element from the set of diversity elements.
[0130] S205. Compare the original text data with at least one sample text data to obtain the comparison result of the original text data and at least one sample text data.
[0131] S206. Based on the comparison results of the at least one diversity element and the original text data and at least one sample text data, obtain the extended constraint conditions corresponding to the original text data.
[0132] S207. Construct the original semantic tree corresponding to the original text data.
[0133] S208. Randomly prune one branch of the original semantic tree and / or swap the positions of two elements under the same branch of the original semantic tree to obtain the transformed semantic tree.
[0134] S209. Input the extended constraints and the text data corresponding to the transformed semantic tree into a pre-trained text data extension model, and obtain the extended text data corresponding to the initial text data output by the text data extension model.
[0135] S210. Obtain the degree of restoration of the extended text data relative to the original text data.
[0136] In this application's embodiments, the degree of restoration refers to the similarity between the generated extended text data and the original text data, or the degree to which the original semantics are preserved. If the generated extended text data is highly consistent with the original text data in terms of semantics and syntax, then the degree of restoration can be considered high. Maintaining the degree of restoration is crucial in text data extension tasks because the extended text data needs to preserve the semantic accuracy and consistency of the original text.
[0137] It should be noted that the extended text data corresponding to the initial text data obtained in step S209 may include multiple data. When the extended text data corresponding to the initial text data includes multiple data, step S210 can be performed on each extended text data one by one.
[0138] In some embodiments, obtaining the degree of restoration of the extended text data relative to the original text data includes: obtaining the degree of restoration of the extended text data relative to the original text data based on n-gram.
[0139] Obtaining the restoration degree of the extended text data relative to the original text data based on n-gram includes the following steps ① to ⑤:
[0140] Step ①: Obtain the first element set and the second element set.
[0141] Among them, the first element set is a set composed of 1 to n word tokens of the original text data; the second element set is a set of 1 to n word token groups of the extended text data, and n is an integer greater than 1.
[0142] Exemplarily, taking the original text data as "The weather is really good today" and the extended text data as "The weather is great today", and n = 3 as an example to illustrate the above steps.
[0143] The 1-word tokens (1-gram) of the original text data "The weather is really good today" include: "今 (jin)", "日 (ri)", "天 (tian)", "气 (qi)", "真 (zhen)", "好 (hao)".
[0144] The 2-word tokens (2-gram) of the original text data "The weather is really good today" include: "今日 (jin ri)", "日天 (ri tian)", "天气 (tian qi)", "气真 (qi zhen)", "真好 (zhen hao)".
[0145] The 3-word tokens (3-gram) of the original text data "The weather is really good today" include: "今日天 (jin ri tian)", "日天气 (ri tian qi)", "天气真 (tian qi zhen)", "气真好 (qi zhen hao)".
[0146] Therefore, the first element set is: {今 (jin), 日 (ri), 天 (tian), 气 (qi), 真 (zhen), 好 (hao), 今日 (jin ri), 日天 (ri tian), 天气 (tian qi), 气真 (qi zhen), 真好 (zhen hao), 今日天 (jin ri tian), 日天气 (ri tian qi), 天气真 (tian qi zhen), 气真好 (qi zhen hao)}.
[0147] The 1-word tokens (1-gram) of the extended text data "The weather is great today" include: "今 (jin)", "日 (ri)", "天 (tian)", "气 (qi)", "很 (hen)", "棒 (bang)".
[0148] The 2-word tokens (2-gram) of the extended text data "The weather is great today" include: "今日 (jin ri)", "日天 (ri tian)", "天气 (tian qi)", "气很 (qi hen)", "很棒 (hen bang)".
[0149] The 3-word tokens (3-gram) of the extended text data "The weather is great today" include: "今日天 (jin ri tian)", "日天气 (ri tian qi)", "天气很 (tian qi hen)", "气很棒 (qi hen bang)".
[0150] Therefore, the second element set is: {今 (jin), 日 (ri), 天 (tian), 气 (qi), 很 (hen), 棒 (bang), 今日 (jin ri), 日天 (ri tian), 天气 (tian qi), 气很 (qi hen), 很棒 (hen bang), 今日天 (jin ri tian), 日天气 (ri tian qi), 天气很 (tian qi hen), 气很棒 (qi hen bang)}.
[0151] Step ②: Calculate the intersection of the first element set and the second element set to obtain the third element set.
[0152] Represent the first set of elements as G1, the second set of elements as G2, and the third set of elements as G3. Then, we have:
[0153] G3 = G1 ∩ G2
[0154] Continuing from the above embodiment, the first set of elements is: {今, 日, 天, 气, 真, 好, 今日, 天天, 天气, 气真, 真好, 今日天, 日天气, 天气真, 气真好}, and the second set of elements is: {今, 日, 天, 气, 很, 棒, 今日, 日天, 天气, 气很, 很棒, 今日天, 日天气, 天气很, 气很棒}. Then, the third set of elements is {今, 日, 天, 气, 今日, 天天, 天气, 今日天, 日天气}.
[0155] Step ③: Calculate the ratio of the number of elements in the third set of elements to the number of elements in the first set of elements to obtain the coverage.
[0156] Represent the first set of elements as G1, the third set of elements as G3, and the coverage as P1. Then, we have:
[0157] P1 = |G3| / |G1|
[0158] Continuing from the above example, |G3| = 8 and |G1| = 15, so P1 = 8 / 15.
[0159] Step ④: Calculate the ratio of the number of elements in the third set of elements to the number of elements in the second set of elements to obtain the similarity.
[0160] Represent the second set of elements as G2, the third set of elements as G3, and the similarity as P2. Then, we have:
[0161] P2 = |G3| / |G2|
[0162] Continuing from the above example, |G3| = 8 and |G2| = 15, so P2 = 8 / 15.
[0163] Step ⑤: Calculate the product of the coverage and the similarity to obtain the restoration degree of the extended text data relative to the original text data.
[0164] Represent the coverage as P1, the similarity as P2, and the restoration degree as P. Then, we have:
[0165] P = P1 * P2
[0166] Continuing from the above example, P1 = 8 / 15 and P2 = 8 / 15, so P ≈ 0.302.
[0167] In some other embodiments, obtaining the restoration degree of the extended text data relative to the original text data includes the following steps I and II:
[0168] Step I: Input the original text data and the extended text data into the target sentence encoding model respectively, and obtain the original sentence vector corresponding to the original text data and the extended sentence vector corresponding to the extended text data output by the target sentence encoding model.
[0169] In some embodiments, the target sentence encoding model can be a Sentence-BERT (SBERT) model. The SBERT model is a pre-trained language model based on the Transformer architecture, primarily used for sentence-level natural language processing tasks, such as sentence similarity calculation, text classification, and sentiment analysis. The SBERT model learns the semantic representation of sentences through unsupervised learning on large-scale text. Unlike traditional word vector models, the SBERT model can directly process sentence-level input and output a fixed-length vector representation that captures the semantic information of the sentence.
[0170] Step II: Calculate the cosine values of the original sentence vector and the extended sentence vector to obtain the degree of restoration of the extended text data relative to the original text data.
[0171] S211. Determine whether the degree of restoration is greater than the degree of restoration threshold.
[0172] In this embodiment, the size of the fidelity threshold is not limited. If the fidelity threshold is larger, more extended data will be filtered out, and the consistency requirement of the data will be more stringent. If the fidelity threshold is smaller, more extended data will be retained after filtering, and the data diversity may be richer. Therefore, in practice, the fidelity threshold can be set in a way that balances the consistency requirement and the diversity requirement.
[0173] In step S211 above, if the degree of restoration is not greater than the degree of restoration threshold, then step S212 is executed as follows:
[0174] S212. Discard the extended text data.
[0175] Since the above embodiments, after generating the extended text data corresponding to the original text data, also obtain the degree of restoration of the extended text data relative to the original text data, and discard the extended text data if the degree of restoration of the extended text data relative to the original text data is not greater than the degree of restoration threshold, the above embodiments can further ensure the consistency of the finally obtained extended text data.
[0176] Based on the same inventive concept, as an implementation of the above method, this application embodiment also provides a text data expansion device. This embodiment corresponds to the aforementioned method embodiment. For ease of reading, this embodiment will not repeat the details of the aforementioned method embodiment one by one, but it should be clear that the text data expansion device in this embodiment can correspondingly implement all the contents of the aforementioned method embodiment.
[0177] This application provides a text data expansion device. Figure 3 This is a schematic diagram of the structure of the expansion device for the text data, as shown below. Figure 3 As shown, the text data expansion device 300 includes:
[0178] Acquisition unit 31 is used to acquire the extended constraint conditions corresponding to the original text data;
[0179] Construction unit 32 is used to construct the original semantic tree corresponding to the original text data;
[0180] Transformation unit 33 is used to transform the original semantic tree to obtain a transformed semantic tree;
[0181] The generation unit 34 is used to extend the text data corresponding to the transformed semantic tree based on the extended constraints to generate extended text data corresponding to the original text data.
[0182] As an optional implementation of this application, the acquisition unit 31 is specifically used to determine at least one attribute based on a preset diversity system; acquire attribute information of the at least one attribute corresponding to the original text data; acquire a set of diversity elements of the original text data based on the attribute information of the at least one attribute; and generate extended constraint conditions corresponding to the original text data based on the set of diversity elements.
[0183] As an optional implementation of this application, the acquisition unit 31 is specifically used to randomly select at least one diversity element from the diversity element set; and generate extended constraint conditions corresponding to the original text data based on the at least one diversity element.
[0184] As an optional implementation of this application, the acquisition unit 31 is specifically used to compare the original text data and at least one sample text data to obtain the comparison result of the original text data and at least one sample text data; and to generate extended constraint conditions corresponding to the original text data based on the comparison result of the original text data and at least one sample text data.
[0185] As an optional implementation of this application, the transformation unit 33 is specifically used to randomly prune a branch of the original semantic tree and / or swap the positions of two elements under the same branch of the original semantic tree to obtain the transformed semantic tree.
[0186] As an optional implementation of this application, the generation unit 34 is specifically used to input the extended constraints and the text data corresponding to the transformed semantic tree into a pre-trained text data extension model, and obtain the extended text data corresponding to the initial text data output by the text data extension model.
[0187] As an optional implementation of this application, the generation unit 34 is further configured to, after expanding the text data corresponding to the transformed semantic tree based on the extended constraint conditions to generate extended text data corresponding to the original text data, obtain the degree of restoration of the extended text data relative to the original text data; determine whether the degree of restoration is greater than the degree of restoration threshold; if not, discard the extended text data.
[0188] As an optional implementation of this application, the generation unit 34 is specifically used to obtain a first element set and a second element set, wherein the first element set is a set of 1 to n word groups of the original text data, and the second element set is a set of 1 to n word groups of the extended text data, where n is an integer greater than 1; calculate the intersection of the first element set and the second element set to obtain a third element set; calculate the ratio of the number of elements in the third element set to the number of elements in the first element set to obtain coverage; calculate the ratio of the number of elements in the third element set to the number of elements in the second element set to obtain similarity; and calculate the product of the coverage and the similarity to obtain the restoration degree of the extended text data relative to the original text data.
[0189] As an optional implementation of this application, the generation unit 34 is specifically used to input the original text data and the extended text data into the target sentence encoding model respectively, obtain the original sentence vector corresponding to the original text data and the extended sentence vector corresponding to the extended text data output by the target sentence encoding model; calculate the cosine value of the original sentence vector and the extended sentence vector to obtain the degree of restoration of the extended text data relative to the original text data.
[0190] The text data expansion device provided in this application embodiment can execute the text data expansion method provided in any of the above embodiments. Its implementation principle and technical effect are similar, and will not be described again here.
[0191] Based on the same inventive concept, embodiments of this application also provide an electronic device. Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application, such as... Figure 4 As shown, the electronic device provided in this embodiment includes a memory 401 and a processor 402. The memory 401 is used to store a computer program, and the processor 402 is used to execute the text data extension method provided in the above embodiment when executing the computer program.
[0192] Based on the same inventive concept, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the computing device to implement the text data expansion method provided in the above embodiments.
[0193] Based on the same inventive concept, this application also provides a computer program product that, when run on a computer, enables the computing device to implement the text data expansion method provided in the above embodiments.
[0194] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.
[0195] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.
[0196] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.
[0197] Computer-readable media include both permanent and non-permanent, removable and non-removable storage media. Storage media can store information using any method or technology; the information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media do not include transient computer-readable media, such as modulated data signals and carrier waves.
[0198] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some or all of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.
Claims
1. A method for expanding text data, characterized in that, include: Obtain the extended constraints corresponding to the original text data; Construct the original semantic tree corresponding to the original text data; The original semantic tree is transformed to obtain a transformed semantic tree; The text data corresponding to the transformed semantic tree is expanded based on the extended constraints to generate extended text data corresponding to the original text data.
2. The method according to claim 1, characterized in that, The extended constraints corresponding to the obtained original text data include: At least one attribute is determined based on a pre-defined diversity system; Obtain the attribute information of at least one attribute corresponding to the original text data; Based on the attribute information of at least one attribute, obtain a set of diverse elements of the original text data; Based on the set of diverse elements, the extended constraints corresponding to the original text data are generated.
3. The method according to claim 2, characterized in that, The step of generating extended constraints corresponding to the original text data based on the set of diverse elements includes: At least one diversity element is randomly selected from the set of diversity elements; Based on the at least one diversity element, the extended constraints corresponding to the original text data are generated.
4. The method according to claim 1, characterized in that, The extended constraints corresponding to the obtained original text data include: The original text data and at least one sample text data are compared to obtain the comparison result between the original text data and at least one sample text data. Based on the comparison results between the original text data and at least one sample text data, extended constraints corresponding to the original text data are generated.
5. The method according to claim 1, characterized in that, The transformation of the original semantic tree to obtain a transformed semantic tree includes: The transformed semantic tree is obtained by randomly pruning one branch of the original semantic tree and / or swapping the positions of two elements under the same branch of the original semantic tree.
6. The method according to claim 1, characterized in that, The step of expanding the text data corresponding to the transformed semantic tree based on the extended constraints to generate extended text data corresponding to the original text data includes: The extended constraints and the text data corresponding to the transformed semantic tree are input into a pre-trained text data extension model, and the extended text data corresponding to the original text data output by the text data extension model is obtained.
7. The method according to claim 1, characterized in that, After expanding the text data corresponding to the transformed semantic tree based on the extended constraints to generate expanded text data corresponding to the original text data, the method further includes: Obtain the degree of restoration of the extended text data relative to the original text data; Determine whether the degree of restoration is greater than the degree of restoration threshold; If not, then discard the extended text data.
8. The method according to claim 7, characterized in that, The step of obtaining the degree of restoration of the extended text data relative to the original text data includes: Obtain a first element set and a second element set, wherein the first element set is a set of 1 to n word groups of the original text data, and the second element set is a set of 1 to n word groups of the extended text data, where n is an integer greater than 1; Calculate the intersection of the first set of elements and the second set of elements to obtain the third set of elements; Calculate the ratio of the number of elements in the third element set to the number of elements in the first element set to obtain the coverage. Calculate the ratio of the number of elements in the third element set to the number of elements in the second element set to obtain the similarity. Calculate the product of the coverage and the similarity to obtain the degree of restoration of the extended text data relative to the original text data.
9. The method according to claim 7, characterized in that, The step of obtaining the degree of restoration of the extended text data relative to the original text data includes: The original text data and the extended text data are respectively input into the target sentence encoding model to obtain the original sentence vector corresponding to the original text data and the extended sentence vector corresponding to the extended text data output by the target sentence encoding model; Calculate the cosine values of the original sentence vector and the extended sentence vector to obtain the degree of restoration of the extended text data relative to the original text data.
10. A text data expansion device, characterized in that, include: The acquisition unit is used to acquire the extended constraints corresponding to the original text data. The construction unit is used to construct the original semantic tree corresponding to the original text data; A transformation unit is used to transform the original semantic tree to obtain a transformed semantic tree; The generation unit is used to extend the text data corresponding to the transformed semantic tree based on the extended constraints to generate extended text data corresponding to the original text data.
11. An electronic device, characterized in that, include: A memory and a processor, the memory being used to store a computer program and the processor being used to cause the electronic device to implement the method for expanding text data as described in any one of claims 1-9 when executing the computer program.
12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a computing device, causes the computing device to implement the text data expansion method according to any one of claims 1-9.
13. A computer program product, characterized in that, When the computer program product is run on a computer, the computer enables the computer to implement the text data expansion method according to any one of claims 1-9.