A data processing method and device
By splitting and combining long statements, combining semantic logical relationships and data enhancement technology, the problem of low reliability of existing data enhancement methods is solved, the significant increase in data volume and logical coherence are achieved, and the data quality of natural language processing is improved. .
Patent Information
- Application Number
- CN202110321272.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-03-25
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-03-25
AI Technical Summary
Existing data augmentation methods in the field of natural language processing such as vocabulary replacement, reverse translation, and text surface conversion lead to low reliability of data augmentation and little change.
By splitting long statements in a short sentence, the semantic logical relationships between sub-statements are obtained and combined according to these relationships are combined to increase the number of long statements, and further enhance the data of sub-statements, including character substitution, reverse translation and text Convert.
The data volume is significantly increased, the reliability of data enhancement is improved, and the generated new statements are logically coherent and syntactical.
Smart Images

Figure CN113033211B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a data processing method and device. Background Art
[0002] In the field of natural language processing, insufficient data can cause many problems, so data enhancement is necessary.
[0003] However, the enhanced data obtained by the currently used enhancement schemes such as vocabulary replacement, back translation, and text surface conversion has little change compared to the original data, resulting in low reliability of data enhancement. Summary of the invention
[0004] In view of this, the present application provides a data processing method and device as follows:
[0005] A data processing method, comprising:
[0006] Obtaining target data, wherein the target data includes a plurality of long sentences;
[0007] Splitting the long sentence into short sentences to obtain multiple sub-sentences;
[0008] Obtaining the semantic logical relationship between the sub-sentences;
[0009] The sub-sentences are combined according to the semantic logical relationship to obtain a plurality of new long sentences.
[0010] Preferably, the above method further comprises, before combining the sub-sentences according to the semantic logical relationship to obtain a plurality of new long sentences:
[0011] Data enhancement is performed on the sub-sentences so that the number of the sub-sentences is increased.
[0012] Preferably, in the above method, after obtaining a plurality of new long sentences, the method further comprises:
[0013] Data enhancement is performed on one or more sub-sentences in the new long sentence, so that the number of the new long sentences is increased.
[0014] In the above method, preferably, data enhancement is performed on the sub-sentence, including any one or more of the following:
[0015] Performing character replacement on the sub-statement;
[0016] Back-translating the sub-sentences;
[0017] Perform text conversion on the substatement.
[0018] Preferably, the above method further comprises, before combining the sub-sentences according to the semantic logical relationship to obtain a plurality of new long sentences:
[0019] According to the semantic logical relationship, the sub-sentences are grouped to obtain a plurality of statement groups;
[0020] According to the semantic logical relationship, the sub-sentences are combined to obtain a plurality of new long sentences, including:
[0021] According to the semantic logical relationship, any one of the sub-sentences is selected from any plurality of the sentence groups for combination to obtain a plurality of new long sentences.
[0022] In the above method, preferably, the semantics of the sub-sentences in the same statement group satisfy a semantic similarity condition.
[0023] The above method preferably selects any one of the sub-sentences from any plurality of the sentence groups according to the semantic logical relationship to combine them to obtain a plurality of new long sentences, including:
[0024] According to the semantic logical relationship, one or more statement sets are obtained, wherein the statement set includes multiple statement groups, and sub-statements included in the statement groups in the same statement set have semantic logical relationships;
[0025] For the statement set, any one of the sub-statements is selected from the statement group included in the statement set for combination to obtain multiple new long sentences, wherein the new long sentences contain multiple sub-statements, and the sub-statements included in the new long sentences have semantic logical relationships.
[0026] The above method preferably obtains one or more sets of statements according to the semantic logical relationship, including:
[0027] According to the semantic logical relationship, a graph corresponding to the statement group is established, wherein the graph includes a plurality of nodes, at least one of the nodes corresponds to the statement group, and at least two of the nodes are connected by a line, and the line between the nodes represents the semantic logical relationship between the statement groups corresponding to the connected nodes;
[0028] One or more statement sets are obtained according to the nodes and the lines in the graph, wherein the statement set includes a plurality of statement groups, and the lines exist between the corresponding nodes in the graph among the statement groups in the same statement set.
[0029] Preferably, the above method further comprises, before splitting the long sentence into short sentences to obtain a plurality of sub-sentences:
[0030] Data enhancement is performed on the long sentences in the target data so that the number of the long sentences in the target data is increased.
[0031] A data processing device, comprising:
[0032] A data acquisition unit, used for acquiring target data, wherein the target data includes a plurality of long sentences, and the long sentences are sentences with complete semantics;
[0033] A sentence splitting unit, used for splitting the long sentence into short sentences to obtain multiple sub-sentences;
[0034] A logic acquisition unit, used to obtain the semantic logical relationship between the sub-sentences;
[0035] The sentence combining unit is used to combine the sub-sentences according to the semantic logical relationship to obtain a plurality of new long sentences.
[0036] From the above scheme, it can be seen that in a data processing method and device provided by the present application, after splitting a long sentence, the sub-sentences are recombined according to the semantic logical relationship between the sub-sentences, so as to obtain multiple new sentences that are different from the long sentences before splitting, thereby increasing the number of long sentences. It can be seen that compared with the situation where the data obtained by the schemes such as vocabulary replacement, reverse translation, and text surface conversion does not change much compared with the original data, the present application can substantially increase the number of sentences, thereby greatly increasing the amount of data, realizing data enhancement, and thereby achieving the purpose of improving the reliability of data enhancement. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0038] Figure 1 A flowchart of a data processing method provided in Example 1 of the present application;
[0039] Figure 2-Figure 6 They are respectively another flow chart of a data processing method provided in Embodiment 1 of the present application;
[0040] Figure 7 A partial flow chart of a data processing method provided in Example 1 of the present application;
[0041] Figure 8-Figure 12 They are respectively example diagrams of statement processing in the embodiments of the present application;
[0042] Fig.13 A schematic diagram of the structure of a data processing device provided in Embodiment 2 of the present application;
[0043] Fig.14 Another structural schematic diagram of a data processing device provided in Embodiment 2 of the present application;
[0044] Fig.15 A schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present application;
[0045] Fig.16 This is an example diagram of a solution using instance cross-enhancement in the prior art;
[0046] Figure 17-Figure 20 They are respectively example diagrams of embodiments of the present application applicable to enhancing training data for deep learning. DETAILED DESCRIPTION
[0047] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0048] refer to Figure 1 , is a flow chart of a data processing method provided in the first embodiment of the present application, which can be applied to electronic devices capable of data processing, such as computers or servers, etc. The technical solution in this embodiment is mainly used to improve the reliability of data enhancement.
[0049] Specifically, the method in this embodiment may include the following steps:
[0050] Step 101: Obtain target data.
[0051] The target data may be training data for natural language processing, such as sample data for model training, etc. In this embodiment, the target data to be enhanced may be obtained by crawling a web page or by other means.
[0052] Specifically, the target data contains multiple long sentences. For example, a text paragraph contains multiple long sentences. Here, a long sentence can be understood as a sentence with complete semantics, such as a sentence ending with a period or a question mark: "My phone won't turn on, I want to send it for repair" and "I broke my phone, how can I repair it?" and so on.
[0053] Step 102: split the long sentence into short sentences to obtain multiple sub-sentences.
[0054] In this embodiment, a long sentence can be split into short sentences by a preset splitting method.
[0055] In one implementation, the present embodiment can split a long sentence into short sentences according to the position of punctuation marks in the long sentence. For example, the present embodiment splits a long sentence according to the position of short sentence connection symbols such as commas, colons, and semicolons in the long sentence, such as splitting "I broke my mobile phone, how can I fix it?" to obtain sub-sentences: "I broke my mobile phone" and "How can I fix it?".
[0056] In another implementation, in this embodiment, a long sentence can be split into short sentences according to the position of the conjunctions in the long sentence. For example, in this embodiment, the long sentence is split according to the position of conjunctions such as "but", "and", "and" and "then" in the long sentence, which represent transitions or parallel meanings, such as "I dropped my phone, and then the phone could not be turned on" is split to obtain sub-sentences: "I dropped my phone" and "the phone could not be turned on".
[0057] Step 103: Obtain the semantic logical relationship between the sub-sentences.
[0058] Among them, semantic logical relations refer to the semantic associations between sub-sentences, such as synonymous relations, conditional relations, causal relations, parallel relations, generalization relations, etc. Among them, synonymous relations can be understood as the events that the two sub-sentences want to express semantically are consistent; conditional relations can be understood as the event semantically expressed by one sub-sentence is the basis for the occurrence of the event semantically expressed by the other sub-sentence; causal relations can be understood as the relationship between the cause and effect of the events semantically expressed by the two sub-sentences; parallel relations can be understood as the parallel relationship between the events semantically expressed by the two sub-sentences; and generalization relations can be understood as the event semantically expressed by one sub-sentence is the superordinate concept or general description of the event semantically expressed by the other sub-sentence.
[0059] Specifically, in this embodiment, the sub-sentences may be semantically analyzed to obtain the events semantically expressed by each sub-sentence, and then obtain the semantic logical relationship between the sub-sentences. The semantic logical relationship here can be understood as the semantic chapter relationship.
[0060] It should be noted that among all the sub-sentences split from the target data, there may be sub-sentences with semantic logical relationships, or there may be sub-sentences without semantic logical relationships. For example, there is a causal semantic logical relationship between the sub-sentences "I dropped my phone" and "the phone can't be turned on", while there is no semantic logical relationship between the sub-sentences "I dropped my phone" and "**'s cold noodles are super delicious".
[0061] Step 104: According to the semantic logical relationship, the sub-sentences are combined to obtain a plurality of new long sentences.
[0062] Among them, in this embodiment, sub-sentences with semantic logical relationships can be combined in a specific combination method, thereby combining multiple new long sentences, thereby increasing the number of long sentences and achieving data enhancement.
[0063] It can be seen from the above scheme that in a data processing method provided in Example 1 of the present application, after splitting a long sentence, the sub-sentences are recombined according to the semantic logical relationship between the sub-sentences, thereby obtaining multiple new sentences that are different from the long sentences before splitting, thereby increasing the number of long sentences. It can be seen that compared with the situation where the data obtained by the schemes such as vocabulary replacement, reverse translation, and text surface conversion does not change much compared to the original data, the present embodiment can substantially increase the number of sentences, thereby greatly increasing the amount of data, realizing data enhancement, and thereby achieving the purpose of improving the reliability of data enhancement.
[0064] In one implementation, before combining the sub-sentences according to the semantic logical relationship in step 104 to obtain a plurality of new long sentences, the method in this embodiment may further include the following steps: Figure 2 As shown in:
[0065] Step 105: Perform data enhancement on the sub-sentences so that the number of sub-sentences is increased.
[0066] Specifically, in this embodiment, data enhancement processing can be performed on sub-sentences once or multiple times in a variety of ways to increase the number of sub-sentences. On this basis, after the sub-sentences are combined according to the semantic logical relationship between the sub-sentences, the number of new long sentences obtained will be further increased.
[0067] For example, in this embodiment, the data enhancement processing of the sub-statement may include any one or more of the following:
[0068] Perform character replacement on sub-statements, reverse translation on sub-statements, and text conversion on sub-statements.
[0069] Among them, character replacement of sub-sentences means replacing keywords in sub-sentences, such as subject or object, with words of the same or similar meanings. The resulting new sub-sentences are semantically the same or similar to the atomic sentence, but the description method is different.
[0070] Back-translation of a sub-statement means translating the sub-statement into a certain language type and then back-translating it into the current language type. The resulting new sub-statement is semantically the same as the atomic statement, but the description method is different.
[0071] The text transformation of a sub-statement refers to the text surface transformation of a sub-statement to obtain a new sub-statement. The obtained new sub-statement is semantically the same as the atomic statement, but the text composition structure is changed.
[0072] It should be noted that step 105 may be performed before step 103. Figure 2 As shown in , alternatively, step 105 can also be performed after step 103, and the resulting technical solutions are all within the protection scope of this application.
[0073] In one implementation, after obtaining a plurality of new long sentences in step 104, the method in this embodiment may further include the following steps: Figure 3 As shown in:
[0074] Step 106: Perform data enhancement on one or more sub-sentences in the new long sentence, so that the number of new long sentences is increased.
[0075] Specifically, in this embodiment, one or more sub-sentences of a new long sentence may be subjected to one or more data enhancement processes in various ways, thereby increasing the number of long sentences. For example, in the first data enhancement process, only one of the sub-sentences in the long sentence is enhanced, thereby increasing the number of long sentences; in the second data enhancement process, two or more sub-sentences in the long sentence may be enhanced, thereby increasing the number of long sentences again, and so on.
[0076] For example, in this embodiment, the data enhancement processing of the sub-sentences in the long sentence may include any one or more of the following:
[0077] Perform character replacement on sub-statements, reverse translation on sub-statements, and text conversion on sub-statements.
[0078] It should be noted that the long sentence subjected to data enhancement in step 106 may include not only a new long sentence obtained after the sub-sentences are combined, but also the original long sentence before being split into sub-sentences.
[0079] In one implementation, in step 102, before splitting the long sentence into short sentences to obtain multiple sub-sentences, the following processing may also be included, such as Figure 4 As shown in:
[0080] Step 107: Perform data enhancement on the long sentences in the target data so that the number of long sentences in the target data is increased.
[0081] Specifically, in this embodiment, data enhancement processing can be performed on long sentences once or multiple times in a variety of ways to increase the number of long sentences. On this basis, the long sentences are split into sub-sentences, and the sub-sentences are combined according to the semantic logical relationship between the sub-sentences. In this way, sentence splitting and reorganization are achieved on the basis of the long sentences whose number has been increased, thereby further increasing the number of long sentences.
[0082] For example, in this embodiment, the data enhancement processing of the long sentence may include any one or more of the following: character replacement of the long sentence, reverse translation of the long sentence, and text conversion of the long sentence.
[0083] Based on the above implementation, in this embodiment, data enhancement can be first performed on the long sentences in the target data. After the sub-sentences are split in step 102, before the sub-sentences are combined according to the semantic logical relationship in step 104 to obtain multiple new long sentences, data enhancement is performed on each sub-sentence respectively, so that the number of sub-sentences is increased. Then, as shown in FIG. Figure 5 As shown in , on the basis of increasing the number of sub-sentences, the sub-sentences are combined according to the semantic logical relationship between the sub-sentences to obtain multiple new long sentences. Finally, data enhancement is performed again on these original long sentences and one or more sub-sentences contained in the new long sentences obtained in step 104, thereby further increasing the number of long sentences, achieving effective data enhancement, and improving the reliability of data enhancement.
[0084] In one implementation, before combining the sub-sentences according to the semantic logical relationship in step 104 to obtain a plurality of new long sentences, the method in this embodiment may further include the following steps: Figure 6 As shown in:
[0085] Step 108: Group the sub-sentences according to the semantic logical relationship to obtain multiple statement groups.
[0086] In this embodiment, the sub-sentences may be grouped according to the semantic logical relationship between the sub-sentences and in a specific grouping manner, so that the sub-sentences that meet the corresponding conditions are divided into the same statement group.
[0087] Specifically, in this embodiment, the sub-sentences with semantic logical relationships can be divided into statement groups according to whether the events semantically expressed by the sub-sentences are the same or similar, so that the semantics of the sub-sentences in the same statement group meet the semantic similarity condition. The semantic similarity condition here can be understood as the similarity of the events semantically expressed between the sub-sentences being greater than a preset similarity threshold. For example, the sub-sentence "the mobile phone cannot be turned on" and the sub-sentence "the mobile phone cannot be turned on" both express the event that the mobile phone cannot be turned on. Therefore, the semantics of these two statements meet the semantic similarity condition and can be divided into the same statement group.
[0088] Based on this, in step 104 of this embodiment, when combining sub-sentences according to semantic logical relationships to obtain multiple new long sentences, it can be specifically implemented in the following manner:
[0089] According to the semantic logical relationship, any sub-sentence is selected from any number of sentence groups for combination to obtain multiple new long sentences.
[0090] That is to say, in this embodiment, sub-sentences are first clustered according to the logical relationship of the sentences, and the sub-sentences that meet the sentence similarity conditions are divided into the same sentence group. The divided sentence groups have the same semantic logical relationship as the sub-sentences contained in them. On this basis, according to the semantic logical relationship between the sentence groups, sub-sentences are arbitrarily selected from these sentence groups and arranged and combined in a specific way, so as to obtain multiple new long sentences, thereby increasing the number of long sentences.
[0091] In a specific implementation, in step 104, when any sub-statement is selected from any multiple statement groups for combination according to the semantic logical relationship, it can be implemented specifically through the following process, such as Figure 7 As shown in:
[0092] Step 701: Obtain one or more statement sets based on semantic logical relationships.
[0093] The statement set includes multiple statement groups, and the sub-statements included in the statement groups in the same statement set have semantic logical relationships.
[0094] It should be noted that different statement sets may contain one or more identical statement groups. Of course, different statement sets must contain one or more different statement groups. The semantic logical relationship between the statement groups contained in the statement set is such that the sub-statements in the statement group can be combined into a long statement, such as Figure 8As shown in , the statement group corresponding to the event "mobile phone purchase" and the statement group corresponding to the event "mobile phone broken" form a statement set, and there is a conditional semantic logical relationship between the statement group of the event "mobile phone purchase" and the statement group of the event "mobile phone broken"; Fig. 9 As shown in , the statement group corresponding to the event "mobile phone purchase", the statement group corresponding to the event "mobile phone broken", and the statement group corresponding to the event "mobile phone sent for repair" constitute a statement set. The statement group of the event "mobile phone purchase" and the statement group of the event "mobile phone broken" have a conditional semantic logical relationship, the statement group corresponding to the event "mobile phone broken" and the statement group corresponding to the event "mobile phone sent for repair" have a causal semantic logical relationship, and so on.
[0095] Specifically, in this embodiment, one or more statement sets may be obtained in the following manner:
[0096] First, based on the semantic logical relationship, a graph corresponding to the sentence groups is established.
[0097] Among them, the graph contains multiple nodes, at least one node corresponds to a statement group, and there is a line between at least two nodes, and the line between the nodes represents the semantic logical relationship between the statement groups corresponding to the connected nodes. Because the included sub-sentences have semantic logical relationships, the statement groups divided in this embodiment also have corresponding semantic logical relationships. Based on this, after the statement groups are used as nodes in the graph, the lines between the nodes are established according to the semantic logical relationships between the statement groups. The established lines between the nodes can actually be understood as the association relationship between the nodes or the node paths between the nodes. The lines between the nodes have a connection direction based on the association relationship or node path represented by them, and the direction of the line represents the direction between the nodes in the association relationship or node path. In this embodiment, the connection between the nodes represents that the statement groups corresponding to the nodes have a semantic logical relationship, and the direction of the line between the nodes represents the type of semantic logical relationship between the statement groups corresponding to the nodes. For example, the line between the node corresponding to the statement group of "buying a mobile phone" and the node corresponding to the statement group of "the mobile phone is broken" points from the node corresponding to the statement group of "buying a mobile phone" to the node corresponding to the statement group of "the mobile phone is broken", indicating that the statement group of "buying a mobile phone" and the statement group of "the mobile phone is broken" have a conditional semantic logical relationship, and the node corresponding to the statement group of "buying a mobile phone" is a conditional node of the node corresponding to the statement group of "the mobile phone is broken", and so on.
[0098] like Fig.10As shown in the graph in , each statement group corresponds to the event expressed by the sub-sentences in the statement group, such as "mobile phone purchase", "mobile phone dropped", "mobile phone soaked in water", "unable to turn on", "mobile phone sent for repair", etc. Due to the semantic logical relationship between the sub-sentences, the statement groups have corresponding semantic logical relationships. Based on this, a graph composed of nodes corresponding to the statement groups is established according to the semantic logical relationship, wherein the statement group of "mobile phone purchase", the statement group of "mobile phone dropped", and the statement group of "mobile phone sent for repair" correspond to a node A, B, and C respectively, and there are lines between A and B and between B and C. The line between A and B represents the semantic logical relationship of the conditional relationship between the statement group of "mobile phone purchase" and the statement group of "mobile phone dropped", and the line between B and C represents the semantic logical relationship of the causal relationship between the statement group of "mobile phone dropped" and the statement group of "mobile phone sent for repair".
[0099] Afterwards, one or more statement sets are obtained based on the nodes and links in the graph.
[0100] The statement set includes multiple statement groups, and there are connections between corresponding nodes in the graph for statement groups in the same statement set.
[0101] Specifically, in this embodiment, according to the connection between the nodes in the graph, the statement groups corresponding to one or more connected nodes are divided into a statement set, so that the nodes corresponding to the statement groups divided into the same statement set are directly connected or connected through other nodes. Moreover, since when dividing the statement set, the statement groups corresponding to all the connected nodes are not divided into one statement set, but the statement groups corresponding to two or more connected nodes are selected from all the connected nodes for set division, therefore, there may be the same statement groups in different statement sets.
[0102] by Fig.10 For example, among the connected nodes A, B, and C, we first select the statement group corresponding to A and B to divide into a statement set x, and then select the statement group corresponding to A, B, and C to divide into a statement set y, such as Fig.11 As shown in , therefore, the statement set x and the statement set y contain the same statement groups: the statement groups corresponding to A and B, and of course different statement groups: the statement group corresponding to C.
[0103] Step 702: For the statement set, any sub-statement is selected from the statement groups included in the statement set for combination to obtain a plurality of new long statements.
[0104] The new long sentence includes multiple sub-sentences, and the sub-sentences included in the new long sentence have semantic logical relations.
[0105] That is to say, in this embodiment, one sub-statement is selected from each statement group included in the statement set, and each sub-statement included in the obtained new long sentence comes from each statement group included in the statement set; when selecting sub-statements from multiple statement groups in the statement set, the sub-statement selection can be performed according to a random algorithm or a permutation and combination method, so as to avoid duplication in the newly combined long sentences, while ensuring that the number of newly combined long sentences is the largest.
[0106] by Fig.11 For example, the statement set y divided from the example, there are 2 sub-statements in the statement group of "Mobile phone purchase", 3 sub-statements in the statement group of "Mobile phone broken", and 3 sub-statements in the statement group of "Mobile phone sent for repair". When selecting sub-statements in the statement set y to form a long statement, we can select any sub-statement from the statement group of "Mobile phone purchase", any sub-statement from the statement group of "Mobile phone broken", and any sub-statement from the statement group of "Mobile phone sent for repair", such as Fig.12 As shown in , after multiple selections, 2*3*3, or 18 long sentences, can be obtained, thereby greatly increasing the number of long sentences.
[0107] refer to Fig.13 , is a schematic diagram of the structure of a data processing device provided in Embodiment 2 of the present application, and the device can be configured in an electronic device capable of data processing, such as a computer or a server, etc. The technical solution in this embodiment is mainly used to improve the reliability of data enhancement.
[0108] Specifically, the device in this embodiment may include the following units:
[0109] The data acquisition unit 1301 is used to acquire target data, where the target data includes a plurality of long sentences, and the long sentences are sentences with complete semantics;
[0110] A sentence splitting unit 1302 is used to split a long sentence into short sentences to obtain multiple sub-sentences;
[0111] A logic acquisition unit 1303, used to obtain the semantic logic relationship between sub-sentences;
[0112] The sentence combining unit 1304 is used to combine the sub-sentences according to the semantic logical relationship to obtain multiple new long sentences.
[0113] It can be seen from the above scheme that in a data processing device provided in the second embodiment of the present application, after splitting a long sentence, the sub-sentences are recombined according to the semantic logical relationship between the sub-sentences, thereby obtaining multiple new sentences that are different from the long sentences before splitting, thereby increasing the number of long sentences. It can be seen that compared with the situation where the data obtained by the schemes such as vocabulary replacement, reverse translation, and text surface conversion does not change much compared to the original data, the present embodiment can substantially increase the number of sentences, thereby greatly increasing the amount of data, realizing data enhancement, and thereby achieving the purpose of improving the reliability of data enhancement.
[0114] In one implementation, the device in this embodiment may include the following units: Fig.14 As shown in:
[0115] The sentence enhancement unit 1305 is used to perform data enhancement on the sub-sentences before the sentence combination unit 1304 combines the sub-sentences according to the semantic logical relationship to obtain multiple new long sentences, so that the number of the sub-sentences is increased.
[0116] In one implementation, the sentence enhancement unit 1305 is further used to perform data enhancement on one or more sub-sentences in the new long sentences after the sentence combination unit 1304 obtains multiple new long sentences, so that the number of the new long sentences is increased.
[0117] Furthermore, the sentence enhancement unit 1305 is specifically used to: perform character replacement on the sub-sentence, perform reverse translation on the sub-sentence, and perform text conversion on the sub-sentence.
[0118] In one implementation, before the statement combination unit 1304 combines the sub-sentences according to the semantic logical relationship to obtain multiple new long sentences, it is also used to: group the sub-sentences according to the semantic logical relationship to obtain multiple statement groups; wherein the semantics of the sub-sentences in the same statement group meet the semantic similarity condition.
[0119] Based on this, the sentence combination unit 1304 is specifically used to: select any one of the sub-sentences from any multiple sentence groups for combination according to the semantic logical relationship, so as to obtain multiple new long sentences.
[0120] In one implementation, the statement combination unit 1304 is specifically used to: obtain one or more statement sets according to the semantic logical relationship, wherein the statement set contains multiple statement groups, and the sub-sentences contained in the statement groups in the same statement set have a semantic logical relationship; for example, establish a graph corresponding to the statement group according to the semantic logical relationship, wherein the graph contains multiple nodes, at least one of the nodes corresponds to the statement group, and there is a connection between at least two of the nodes, and the connection between the nodes represents the semantic logical relationship between the statement groups corresponding to the connected nodes; obtain one or more statement sets according to the nodes and the connections in the graph, wherein the statement set contains multiple statement groups, and the connection exists between the corresponding nodes in the graph between the statement groups in the same statement set; then, for the statement set, arbitrarily select one of the sub-sentences from the statement groups contained in the statement set for combination to obtain multiple new long sentences, wherein the new long sentences contain multiple sub-sentences, and there is a semantic logical relationship between the sub-sentences contained in the new long sentences.
[0121] In one implementation, the sentence enhancement unit 1305 is also used to perform data enhancement on the long sentences in the target data before the sentence splitting unit 1302 splits the long sentence into short sentences to obtain multiple sub-sentences, so that the number of long sentences in the target data is increased.
[0122] It should be noted that the specific implementation of each unit in this embodiment can refer to the corresponding content in the previous text and will not be described in detail here.
[0123] refer to Fig.15 , is a schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present application, and the electronic device may be an electronic device capable of data processing, such as a computer or a server, etc. The technical solution in this embodiment is mainly used to improve the reliability of data enhancement.
[0124] Specifically, the electronic device in this embodiment may include the following structure:
[0125] The memory 1501 is used to store the application program and the data generated by the operation of the application program;
[0126] Processor 1502 is used to execute an application program to achieve: obtaining target data, wherein the target data includes multiple long sentences; splitting the long sentences into short sentences to obtain multiple sub-sentences; obtaining semantic logical relationships between the sub-sentences; and combining the sub-sentences according to the semantic logical relationships to obtain multiple new long sentences.
[0127] The electronic device in this embodiment may also include other components, such as a display, input and output devices, etc.
[0128] It can be seen from the above scheme that in an electronic device provided in the third embodiment of the present application, after splitting a long sentence, the sub-sentences are recombined according to the semantic logical relationship between the sub-sentences, thereby obtaining multiple new sentences that are different from the long sentences before splitting, thereby increasing the number of long sentences. It can be seen that compared with the situation where the data obtained by the schemes such as vocabulary replacement, reverse translation, and text surface conversion does not change much compared to the original data, the present embodiment can substantially increase the number of sentences, thereby greatly increasing the amount of data, realizing data enhancement, and thereby achieving the purpose of improving the reliability of data enhancement.
[0129] Taking the training of a deep learning model as an example, the technical solution of this application is described in detail:
[0130] First, the inventors of this application discovered during their research on deep learning models that currently in the field of natural language processing, insufficient data can lead to many problems, such as insufficient training corpus making deep learning models useless, and insufficient test data making it impossible to measure the true performance of the model. Data augmentation technology plays a key role in providing timely assistance. The existing solutions are as follows:
[0131] 1. Vocabulary replacement, reverse translation, and text surface conversion schemes, but this implementation does not increase the diversity of the data and does not change much compared to the original data;
[0132] 2. The random noise injection scheme, but this implementation scheme may have a negative impact on the original semantics after adding noise;
[0133] 3. The solution of instance cross-enhancement, but the sentences generated in this solution are not logically coherent and do not meet the grammatical or semantic requirements, such as Fig.16 as shown in .
[0134] In view of this, the inventors of the present application found through analyzing existing solutions that they did not really increase the data scale and sometimes might even do more harm than good. Therefore, a data enhancement solution based on the logical relationship between events was proposed.
[0135] In the technical solution of this application, combined with the event graph, the events are first combined according to the logical relationship between the events, and then the description data of each event is enhanced. In this way, data diversity can be enhanced and the logic of generated data can be guaranteed to be correct. The specific technical solution is as follows:
[0136] 1. First, extract and construct the event graph, which includes the following processing:
[0137] (1) Crawl or generalize some data;
[0138] (2) Parse each sentence into small clauses;
[0139] (3) Determine the logical relationship between clauses by analyzing the relationship between the text structures;
[0140] (4) Cluster the sentences to obtain a logical graph.
[0141] like Fig.17 As shown in , in this embodiment, multiple long sentences are first crawled, and then small sentences and logical relationships are obtained through disassembly and relationship extraction. After clustering and summarization or extraction, a graph is obtained, such as Fig.18 as shown in .
[0142] 2. Secondly, combine events according to their logical relationships, such as Fig.19 As shown in , according to the logical relationship, multiple statement sets are combined, and there are corresponding logical relationships between the statement groups in each statement set;
[0143] 3. Finally, perform data augmentation (vocabulary replacement, back-translation, etc.) on each event instance, such as Fig. 20 As shown in , statement enhancement is performed on the statement groups in the statement set respectively, so that the statements in the statement groups are increased. Based on this, more long sentences can be obtained after statement combination.
[0144] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.
[0145] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0146] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
[0147] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A data processing method, comprising: Obtaining target data, wherein the target data includes a plurality of long sentences; Splitting the long sentence into short sentences to obtain multiple sub-sentences; Obtaining a semantic logical relationship between the sub-sentences, wherein the semantic logical relationship is a semantic text relationship; According to the semantic logical relationship, the sub-sentences are combined to obtain a plurality of new long sentences; Wherein, before combining the sub-sentences according to the semantic logical relationship to obtain a plurality of new long sentences, the method further includes: Performing data enhancement on the sub-sentences so that the number of the sub-sentences is increased; After obtaining a plurality of new long sentences, the method further includes: Data enhancement is performed on one or more sub-sentences in the new long sentence, so that the number of the new long sentences is increased.
2. According to the method of claim 1, data enhancement is performed on the sub-statement, including any one or more of the following: Performing character replacement on the sub-statement; Back-translating the sub-sentences; Perform text conversion on the substatement.
3. The method according to claim 1, before combining the sub-sentences according to the semantic logical relationship to obtain a plurality of new long sentences, the method further comprises: According to the semantic logical relationship, the sub-sentences are grouped to obtain a plurality of statement groups; According to the semantic logical relationship, the sub-sentences are combined to obtain a plurality of new long sentences, including: According to the semantic logical relationship, any one of the sub-sentences is selected from any plurality of the sentence groups for combination to obtain a plurality of new long sentences.
4. According to the method of claim 3, the semantics of the sub-sentences in the same statement group meet the semantic similarity condition.
5. The method according to claim 3, wherein, according to the semantic logical relationship, any one of the sub-sentences is selected from any plurality of the sentence groups for combination to obtain a plurality of new long sentences, comprising: According to the semantic logical relationship, one or more statement sets are obtained, wherein the statement set includes multiple statement groups, and sub-statements included in the statement groups in the same statement set have semantic logical relationships; For the statement set, any one of the sub-statements is selected from the statement group included in the statement set for combination to obtain multiple new long sentences, wherein the new long sentences contain multiple sub-statements, and the sub-statements included in the new long sentences have semantic logical relationships.
6. The method according to claim 5, obtaining one or more sets of statements according to the semantic logical relationship, comprises: According to the semantic logical relationship, a graph corresponding to the statement group is established, wherein the graph includes a plurality of nodes, at least one of the nodes corresponds to the statement group, and at least two of the nodes are connected by a line, and the line between the nodes represents the semantic logical relationship between the statement groups corresponding to the connected nodes; One or more statement sets are obtained according to the nodes and the lines in the graph, wherein the statement set includes a plurality of statement groups, and the lines exist between the corresponding nodes in the graph among the statement groups in the same statement set.
7. The method according to claim 1, before splitting the long sentence into short sentences to obtain a plurality of sub-sentences, the method further comprises: Data enhancement is performed on the long sentences in the target data so that the number of the long sentences in the target data is increased.
8. A data processing device, comprising: A data acquisition unit, used for acquiring target data, wherein the target data includes a plurality of long sentences, and the long sentences are sentences with complete semantics; A sentence splitting unit, used for splitting the long sentence into short sentences to obtain multiple sub-sentences; A logic acquisition unit, used to obtain the semantic logical relationship between the sub-sentences; A sentence combining unit, used for combining the sub-sentences according to the semantic logical relationship to obtain a plurality of new long sentences; Wherein, before the sentence combination unit, the device is further used to perform data enhancement on the sub-sentences so that the number of the sub-sentences is increased; After obtaining a plurality of new long sentences, the data processing device further comprises: Data enhancement is performed on one or more sub-sentences in the new long sentence, so that the number of the new long sentences is increased.
Citation Information
Patent Citations
Logic expression determination method, device and equipment and medium
CN109670024A
Method and device for automatically generating article based on template
CN110705244A
Effective data enhancement method for improving neural machine translation effect
CN110852117A
User statement mining and augmentation method and device, terminal and storage medium
CN111159350A
Triple extraction method and device
CN111178079A
