Text regularization method and related device, electronic device, and storage medium

Abstract

The present application discloses a text regularization method and related devices, electronic devices, and storage media, wherein the text regularization method includes: parsing the text to be regularized to obtain a target subtext; wherein the text to be regularized is composed of a plurality of subtexts, the target subtext is the subtext to be regularized, the target subtext needs to be transcribed into a target language, and the grammar of the target language involves gender, number, and case; identifying the attribute categories of the target subtext regarding a plurality of attributes; wherein the plurality of attributes include gender, number, and case attributes, and the attribute categories of the gender, number, and case attributes include the gender, number, and case categories of the target subtext in the target language; based on the attribute categories of the target subtext, transcribe the target subtext into the target language to obtain a regularized subtext having the same semantics as the target subtext; based on the regularized subtext corresponding to the target subtext, obtain a regularized text corresponding to the text to be regularized. The above scheme can improve the accuracy and convenience of text regularization of gender, number, and case languages.

Description

Technical Field

[0001] The present application relates to the technical field of natural speech processing, and in particular to a text regularization method and related devices, electronic devices, and storage media. Background Art

[0002] In the field of natural language, there are many application scenarios that want to convert non-standard text into standard text in the target language. Taking speech synthesis as an example, in the front-end processing of speech synthesis, irregularly written text such as numbers, time and date, currency units, special symbols, etc. need to be accurately converted into words in the target language, so as to ensure accurate front-end information input in the speech synthesis process and obtain accurate speech synthesis results. Taking Chinese as an example, it is necessary to transcribe "123" into Chinese "one hundred and twenty-three", transcribe "1 / 10" into Chinese "one tenth", transcribe "8:00am" into "eight o'clock in the morning", etc.

[0003] For text regularization of general languages, one method is to transcribe the text using rules. Specifically, certain transcription rules are set in advance, and when the text matches the transcription rules, the text is regularized according to the transcription rules. The other method is to transcribe the text using an end-to-end model. Specifically, after the text is input into the end-to-end model, the end-to-end model directly outputs the regularized text using machine translation.

[0004] However, in the language environment involving gender, number and form, text with the same semantics may have different writing methods and pronunciations in different genders and numbers. Simply using end-to-end models or rules to transcribe text cannot guarantee accuracy. On the other hand, due to the diversity of gender, number and form language conversion, there are more stringent requirements on the quality and quantity of training data. Therefore, how to improve the accuracy and convenience of text regularization in gender, number and form languages ​​has become an urgent problem to be solved. Summary of the invention

[0005] The main technical problem solved by the present application is to provide a text regularization method and related devices, electronic devices, and storage media, which can improve the accuracy and convenience of text regularization of numerical language.

[0006] In order to solve the above technical problems, the first aspect of the present application provides a text regularization method, comprising: parsing a text to be regularized to obtain a target sub-text; wherein the text to be regularized is composed of a plurality of sub-texts, the target sub-text is the sub-text to be regularized, the target sub-text needs to be transcribed into a target language, and the grammar of the target language involves gender, number and case; identifying attribute categories of the target sub-text regarding a plurality of attributes; wherein the plurality of attributes include gender, number and case attributes, and the attribute categories of the gender, number and case attributes include the gender, number and case categories of the target sub-text in the target language; based on the attribute categories of the target sub-text, transcribe the target sub-text into the target language to obtain a regularized sub-text having the same semantics as the target sub-text; based on the regularized sub-text corresponding to the target sub-text, obtain a regularized text corresponding to the text to be regularized.

[0007] In order to solve the above technical problems, the second aspect of the present application provides a text regularization device, including: a parsing module, a recognition module, a transcription module and an acquisition module, the parsing module is used to parse the text to be regularized to obtain a target sub-text; wherein the text to be regularized is composed of a plurality of sub-texts, the target sub-text is the sub-text to be regularized, the target sub-text needs to be transcribed into a target language, and the grammar of the target language involves gender, number and case; the recognition module is used to identify the attribute categories of the target sub-text regarding a plurality of attributes; wherein the plurality of attributes include gender, number and case attributes, and the attribute categories of the gender, number and case attributes include the gender, number and case categories of the target sub-text in the target language; the transcription module is used to transcribe the target sub-text into the target language based on the attribute categories of the target sub-text to obtain a regularized sub-text having the same semantics as the target sub-text; the acquisition module is used to obtain the regularized text corresponding to the text to be regularized based on the regularized sub-text corresponding to the target sub-text.

[0008] In order to solve the above technical problem, the third aspect of the present application provides an electronic device, including a memory and a processor coupled to each other, wherein the memory stores program instructions, and the processor is used to execute the program instructions to implement the text regularization method in the first aspect.

[0009] In order to solve the above technical problem, the fourth aspect of the present application provides a computer-readable storage medium, which stores program instructions that can be executed by a processor, and the program instructions are used to implement the text regularization method in the above first aspect.

[0010] The above scheme parses the text to be regularized, obtains the target sub-text to be regularized, identifies the attribute categories of the target sub-text regarding several attributes, and then transcribes the target sub-text into the target language based on the attribute categories of the target sub-text to obtain a regularized sub-text with the same semantics as the target sub-text. Finally, based on the regularized sub-text corresponding to the target sub-text, the regularized text corresponding to the text to be regularized is obtained. Since this scheme first determines the attribute categories of the target sub-text and then uses the attribute categories to transcribe the text into the target language, it cleverly converts the transcription problem into a classification problem, and implements classification before transcription, thereby improving the accuracy and convenience of text regularization for numerical languages. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is a flow chart of an embodiment of the regularization method of the present application text;

[0012] Figure 2 It is a flow chart of another embodiment of the regularization method of the present application text;

[0013] Figure 3 It is a schematic diagram of a framework of an embodiment of a regularization device of the present application text;

[0014] Figure 4 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application;

[0015] Figure 5 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0016] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0017] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0018] The terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.

[0019] See also Figure 1 , Figure 1 It is a flow chart of an embodiment of the regularization method of the present application text.

[0020] Specifically, the following steps may be included:

[0021] Step S11: parse the text to be regularized to obtain the target subtext.

[0022] The text regularization method of the present application intends to convert the text to be regularized into a text that conforms to the standard form of the target language. The target language can be Arabic, Russian, Polish and other languages ​​whose grammar involves gender and number. In the target language whose grammar involves gender and number, the writing form and pronunciation of nouns, pronouns or adjectives will have corresponding changes due to semantics or position in the text. For example, the writing form of the text will be different if the suffixes before and after the text are different. Therefore, in the language involving gender and number, the same text will have different writing methods and pronunciations for different genders and numbers. Therefore, it is necessary to accurately convert the irregularly written text to be regularized into the text of the target language.

[0023] The text to be regularized is a text that needs to be regularized, and it is composed of several sub-texts. The sub-text can be text components such as numbers, symbols, and words, or any combination thereof, for example, only numbers, only symbols, and a combination of numbers and symbols, etc. If the sub-text is already in the standard form of the target language, it is a sub-text that does not need to be regularized, and if the sub-text is a non-standard form of the target language, it needs to be regularized. Therefore, the present application first parses the text to be regularized to obtain the target sub-text, which is a sub-text that needs to be regularized and needs to be transcribed into the target language. The parsing method of the text to be regularized can be any text parsing method in the prior art, such as word segmentation, neural network model, etc., and is not specifically limited here.

[0024] For example, the target language is Arabic, and there is a text to be regularized: Among them, the sub-texts "D", "3200", "1080", "×", and "1920" are target sub-texts that need to be regularized because they are not in the standard form of Arabic. The remaining sub-texts are already in Arabic and do not need to be regularized.

[0025] Step S12: Identify the attribute categories of the target subtext with respect to several attributes.

[0026] Different subtexts may belong to the same or different attributes, and the attribute categories of different subtexts under the same attribute may also be the same or different. Several attributes include gender, number, case, numeral, and symbol attributes, which are not specifically limited here.

[0027] In the case where the grammar of the target language involves gender, number and case, several attributes include gender, number and case attributes, and the attribute category of the gender, number and case attribute includes the gender, number and case category of the target sub-text in the target language. The gender of the gender, number and case category refers to the fact that nouns, pronouns and adjectives express people's category understanding of the text and its attributes through changes in grammatical forms. The gender of the gender, number and case category may include feminine, masculine and neuter attribute categories. The number of the gender, number and case category refers to the fact that nouns, pronouns, etc. express people's understanding of the quantity of the text through changes in grammatical forms. The number of the gender, number and case category may include singular, plural and dual attribute categories; the case of the gender, number and case category refers to the case position, which is also the relationship between nouns, pronouns, etc. and other text components through changes in grammatical forms. Understandably, the case position representations of different languages ​​are inconsistent. For example, the number '1' in Arabic may correspond to (feminine nominative case), (feminine accusative), (feminine genitive), (masculine nominative), (masculine accusative), (Masculine genitive) There are 6 gender and number categories in total.

[0028] In the case where the target subtext is a number, the several attributes also include a numeral attribute, and the attribute category of the numeral attribute includes the numeral category of the target subtext. In a disclosed embodiment, the numeral category may include cardinal numbers, ordinal numbers, and character strings. For numbers, both the numeral category and the gender and case category are included, for example, cardinal numbers, ordinal numbers, and character strings have three numeral categories and (feminine nominative case), (feminine accusative), (feminine genitive), (masculine nominative), (masculine accusative), (Masculine genitive) Any combination of 6 gender and number categories, a total of 18 categories, such as cardinal number-feminine nominative. The main difficulty in text regularization of gender and number languages ​​is concentrated on the determination of the numeral category of numbers. For digital subtexts such as time and date, they can be classified by combining numbers or by combining the numeral categories of numbers one by one. The specific transcription form of numbers depends on a large number of rules, and because the gender and number need to be determined according to the context in many cases, the accuracy of relying solely on preset text rules or end-to-end models cannot be guaranteed; in addition, due to the diversity of digital conversion forms in gender and number languages, in order to achieve the same effect as general languages, the end-to-end model method has higher requirements on data volume and data quality. In particular, there are multiple transcription methods for a number, and the end-to-end model will face greater pressure when making predictions. Compared with the existing text regularization research, which does not take into account the characteristics of multiple numeral categories when numbers are used as sub-texts, and does not distinguish numbers separately to implement text regularization, the present application can identify the numeral category of numbers and then transcribe the text based on the numeral category, so that it can handle text regularization in which the same numeric sub-text has multiple transcription methods.

[0029] For example, in the text regularization process of the front-end text in speech synthesis, the writing method to be used for pronunciation needs to be determined according to the content before and after the text and the overall semantics to determine its semantic category, so the accuracy of the semantic category of the target sub-text is particularly important. In the case where the target sub-text is a symbol, the several attributes also include symbol attributes, and the attribute categories of symbol attributes include the semantic category of the target sub-text. For example, the semantic category of the symbol "-" can be "minus sign" or "range from... to..."; the semantic category of the symbol ":" can be "colon", "time point", "score"; and currency units and other special symbols also need to determine the semantic category based on the specific context.

[0030] Step S13: based on the attribute category of the target sub-text, transcribe the target sub-text into the target language to obtain a regularized sub-text having the same semantics as the target sub-text.

[0031] For the classified attribute category results, they are transcribed into the text of the target language according to specific rules. For example, after determining the attribute category of the target sub-text, based on the mapping relationship between the attribute category and the text expressed in the target language, the text expressed in the target language corresponding to the attribute category is used as the regularized sub-text, and the target sub-text is transcribed into the target language based on the attribute category of the target sub-text, so as to obtain a regularized sub-text with the same semantics as the target sub-text. Compared with the end-to-end model, the text regularization is essentially translation. This application classifies first and then transcribes, and treats text regularization as a classification task, using less data relative to the end-to-end model, while also avoiding the common irrecoverable errors of the end-to-end model. Among them, the mapping relationship is the corresponding relationship between the sub-text, the attribute category, and the regularized sub-text expressed in the target language.

[0032] In a disclosed embodiment, based on the attribute category of the target subtext, the target subtext is transcribed into the target language to obtain a regularized subtext with the same semantics as the target subtext. The specific steps are as follows: query in the transcription rule set to obtain a first subtext that satisfies the matching condition with the target subtext; wherein the transcription rule set includes a plurality of subtext pairs, the subtext pair includes a first subtext and a second subtext with the same semantics, and the second subtext is expressed in the target language according to the attribute category of the first subtext; the second subtext in the subtext pair to which the first subtext belongs is used as the regularized subtext of the target subtext. The matching condition can be that the first subtext and the target subtext have the same semantics and the attribute category is exactly the same, or the attribute category of the first subtext and the target subtext are the same.

[0033] For text regularization of general languages, the prior art uses preset text rules to transcribe text. Specifically, a certain number of transcription rules are preset, and when the text matches the transcription rules, the text is regularized according to the transcription rules. When using preset text rules for text transcription, the transcription rules must be preset, and a sub-text can only transcribe text in one target language. For example, the regularized sub-text corresponding to the sub-text number '1' is only 'one', and the regularized sub-text corresponding to the sub-text number '2' is only 'two'. However, in languages ​​whose grammar involves gender, number and case, a sub-text often corresponds to multiple forms and needs to be transcribed into multiple regularized sub-texts. For example, when the Arabic sub-text is the number '1', it may correspond to (feminine nominative case), (feminine accusative), (feminine genitive), (masculine nominative), (masculine accusative), (Masculine genitive) 6 different regularized subtexts, each attribute category corresponds to a regularized subtext, at this time, text transcription using preset text rules cannot be achieved, and the disclosed embodiment determines the regularized subtext based on the attribute category of the target subtext, which is suitable for the scenario where the same subtext corresponds to multiple regularized subtexts. Simply relying on preset text rules for text regularization of masculine and numeral language will have the following limitations: First, it is impossible to find universal rules for the numeral category, semantic category, etc. of the subtext, and it is necessary to customize the rules separately, resulting in a large number of rules required; second, if an unfamiliar text is encountered or the corresponding rules are not included, errors often occur, and the case of the text often needs to be determined based on the specific semantics, and it is impossible to find applicable rules, so the generalization and accuracy of the rules are poor; third, when there are multiple transcription methods for the same text, text transcription cannot be performed relying solely on rules. The prior art also uses end-to-end models for text transcription. Specifically, after the text is input into the end-to-end model, the end-to-end model directly outputs the regularized text by machine translation. However, due to the limitations of deep learning networks and the amount of training data, there are often some irrecoverable errors, such as the sub-text of the number '123' being transcribed as "one hundred and thirty-two". The present application first determines the attribute category of the sub-text, and then uses the attribute category to transcribe the text in the target language, performing classification before transcription, which can solve the above-mentioned problem of text transcription that relies solely on rules or end-to-end models.

[0034] Therefore, by identifying the attribute categories of the target sub-text regarding several attributes, and then transcribing the target sub-text into the target language based on the attribute categories of the target sub-text, a regularized sub-text with the same semantics as the target sub-text is obtained. This can solve the problem in the prior art that preset text rules and end-to-end models cannot perform regular transcription of text according to the text context.

[0035] Step S14: Based on the regularized sub-text corresponding to the target sub-text, a regularized text corresponding to the text to be regularized is obtained.

[0036] Since the target subtext is the subtext that needs to be regularized in the text to be regularized, after obtaining the regularized subtext corresponding to the target subtext, the regularized text corresponding to the text to be regularized can be obtained based on the regularized subtext corresponding to the target subtext. For example, when the regularized text corresponding to the text to be regularized is obtained based on the regularized subtext corresponding to the target subtext, the target subtext in the text to be regularized is replaced with the regularized text corresponding to the target subtext, and the regularized text can be obtained.

[0037] The above scheme parses the text to be regularized, obtains the target sub-text to be regularized, identifies the attribute categories of the target sub-text regarding several attributes, and then transcribes the target sub-text into the target language based on the attribute categories of the target sub-text to obtain a regularized sub-text with the same semantics as the target sub-text, and finally obtains the regularized text corresponding to the text to be regularized based on the regularized sub-text corresponding to the target sub-text. Since this scheme first determines the attribute categories of the target sub-text and then uses the attribute categories to transcribe the text into the target language, it implements classification before transcription, thereby improving the accuracy and convenience of text regularization for numerical languages.

[0038] Text regularization for gender, number, and case languages ​​mainly involves determining the gender (feminine, masculine, and neutral), number (singular, plural, and dual), and case attributes of non-standard target subtexts in context, such as gender, number, and case categories, numeral categories, and semantic categories. The transcription form is then determined by the attribute categories to obtain a regularized subtext with the same semantics as the target subtext, thereby obtaining a regularized text in the target language by first classifying and then transcribing. In order to further improve the accuracy and convenience of text regularization for gender, number, and case languages, the disclosed embodiment can use an attribute classification network to determine the attribute categories. For details, please refer to Figure 2 , Figure 2 This is a flow chart of another embodiment of the text regularization method of the present application. Specifically, it may include the following steps:

[0039] Step S21: Analyze the regular text to obtain a number of sub-texts.

[0040] The analysis method of the text to be regularized can be any existing implementation method, which is not specifically limited here. By analyzing the text to be regularized, a plurality of subtexts are obtained.

[0041] Step S22: using the regular rule set of the target language, respectively perform rule matching on a plurality of subtexts to obtain a target subtext.

[0042] The regular rule set includes at least one of the first subset and the second subset, wherein the first subset includes several text rules that do not require regular processing, and the second subset includes several text rules that require regular processing. For example, in a disclosed embodiment, the regular rule set includes the first subset and the second subset, and several subtexts are matched with the first subset and the second subset by rules, and the subtexts that successfully match the second subset are used as target subtexts, and the subtexts that successfully match the first subset are discarded and not used as target subtexts. In the case where the regular rule set only includes the first subset, several subtexts are matched with the first subset by rules, and the subtexts that fail to match are used as target subtexts. In the case where the regular rule set only includes the second subset, several subtexts are matched with the second subset by rules, and the subtexts that successfully match are used as target subtexts. The regular rule set is a basic regular rule set established for the target language, and the set includes certain text rules, that is, the text rules can uniquely determine whether the subtext needs to be regular processed. The text rules in the first subset and the second subset can be customized and are not specifically limited here.

[0043] Therefore, when parsing the text to be regularized and obtaining the target sub-text, the text to be regularized can be analyzed to obtain several sub-texts, and then the regular rule set of the target language is used to match the rules of the several sub-texts respectively to obtain the target sub-text. The sub-texts corresponding to several text rules that do not require regular processing are filtered out in advance, and there is no need to execute subsequent steps S23 to S24, thereby reducing the amount of data processing and improving the efficiency of text regularization.

[0044] Step S23: identifying the attribute categories of the target subtext with respect to several attributes, wherein the attribute categories of the several attributes are predicted by processing the regular text using an attribute classification network.

[0045] In the disclosed embodiment, the attribute categories of several attributes are predicted by processing the regularized text using an attribute classification network. Using the attribute classification network to assist in text regularization is not the end-to-end model in the prior art directly outputting the regularized text by machine translation, but rather treating gender, number, case, numeral, and symbol attributes as a category and using the network to predict, and then performing text transcription according to the specific transcription rules in step S24.

[0046] The attribute classification network is trained using sample texts, and the sample subtexts in the sample texts are annotated with sample tags. When the sample subtexts need to be regularized, the sample tags include the sample attribute categories of the sample subtexts regarding several attributes. When the sample subtexts do not need to be regularized, the sample tags are preset tags. Preset tags can be customized, for example, preset tags are <self>The several attributes of the sample subtext include gender, number and case attributes, or any combination of gender, number and case attributes and numeral attributes and symbol attributes. When the attributes of the sample subtext include gender, number and case attributes, the sample attribute category of the sample tag includes the gender, number and case category of the sample subtext in the target language. For example, the number '1' in Arabic corresponds to (feminine nominative case). When the attributes of the sample sub-text include numeral attributes, the sample attribute categories of the sample tag include the numeral category of the sample sub-text in the target language. When the attributes of the sample sub-text include symbol attributes, the sample attribute categories of the sample tag include the semantic category of the sample sub-text in the target language. For example, the semantic category of the symbol "-" is "minus sign". When obtaining the sample text, you can crawl the corresponding sample sub-text in the large text, and then manually annotate the sample sub-text. For sample sub-texts with specific semantics such as time points or dates, you only need to mark them out as a whole. For other numbers, you need to mark the numeral category and gender and number category. For symbols, you also need to mark the semantic category and gender and number category.

[0047] The attribute classification network can be a network such as a recurrent neural network (RNN) as a single-task model for classification; two parallel networks can also be used as a multi-task model, that is, multiple networks classify different attributes respectively to achieve attribute classification in parallel. The main structure of the network can be a two-layer bidirectional long short-term memory (LSTM) network. The target sub-text is used as the input of the attribute classification network, then passes through the embedding layer, and then passes through two layers of bidirectional long short-term memory networks, and finally outputs the attribute category corresponding to the target sub-text. The composition of the attribute classification network can be customized as needed, as long as it can achieve the attribute classification task.

[0048] In order to enhance the performance of the attribute classification network, appropriate auxiliary training information may be added to the network input of the training attribute classification network, for example, the auxiliary training information may be the N-gram character information of the sample subtext, the semantic information of the sample subtext on the pre-trained model, etc. For example, in a disclosed embodiment, the training steps of the attribute classification network include: obtaining a first embedding representation of the sample subtext after vectorization, obtaining the N-gram character information of the sample subtext, and obtaining a second embedding representation of the N-gram character information after vectorization; fusing the first embedding representation and the second embedding representation to obtain a fused embedding representation of the sample subtext; using the attribute classification network to perform classification prediction on the fused embedding representation to obtain a predicted tag of the sample subtext; wherein the predicted tag is a preset tag, or a predicted attribute category of the sample subtext with respect to several attributes; and adjusting the network parameters of the attribute classification network based on the difference between the sample tag and the predicted tag. As another example, in a disclosed embodiment, the sample sub-text is first input into a pre-training model to obtain the semantic information of the sample sub-text on the pre-training model, and then the semantic information of the sample sub-text on the pre-training model is obtained through vectorization of an embedded representation; the embedded representation is classified and predicted using an attribute classification network to obtain a predicted tag of the sample sub-text; similarly, the predicted tag is a preset tag, or a predicted attribute category of the sample sub-text with respect to several attributes; based on the difference between the sample tag and the predicted tag, the network parameters of the attribute classification network are adjusted. Among them, the preset tag can be customized and is not specifically limited here. In order to indicate that the sample sub-text in the sample text does not need to be regularized, the preset tag of the sample sub-text that does not need to be regularized is marked as <self>.

[0049] In a disclosed embodiment, after using the regular rule set of the target language to match the rules of several subtexts respectively, and obtaining the target subtext, and before detecting the attribute categories of various attributes respectively, and obtaining the detection result, the preset text rules of the target language can be used to match the preset text rules of several target subtexts respectively, so that the target subtext that successfully matches the preset text rules is transcribed into a regularized subtext represented in the target language using the preset text rules, so that for the target subtext that can achieve text transcription using the preset text rules, it is not necessary to identify the attribute category, and it is not necessary to perform the subsequent steps S23 to S24, and the preset text rules are directly used for text regularization, and the target subtext that fails to match the preset text rules is input into the attribute classification network. The preset text rules can be various transcription rules pre-set when using the preset text rules to transcribe text in the prior art. For example, the standardized conversion form corresponding to "xx:xx" in the preset text rules is a time point. If the subtext is "08:00", the text regularization result of the subtext can be "eight o'clock" according to the preset text rules. In addition, in order to increase the scalability of features before and after, rule methods based on finite state transition machines and weight-based finite state transition machines have also been developed. For details, please refer to the prior art and will not be described in detail here. Since the use of preset text rules for text regularization requires the formulation of corresponding rules according to the grammatical rules of the target language, most text regularization problems can be solved for languages ​​without gender and number, but there is a problem that more than one possible transcription text cannot be processed, and too many rules may cause defects in maintenance and flexibility. Therefore, the preset text rules and attribute classification networks can be combined to perform text regularization on the sub-texts that need to be regularized. Specifically, for determining the transcribed text directly according to the preset text rules, there is no need to go through the attribute classification network, thereby avoiding the irrecoverable errors caused by the limitations of the deep learning network and the amount of training data as mentioned above, and reducing the impact caused by classification errors. For sub-texts that cannot be regularized using preset text rules, the attribute classification network is used to process the regular text and predict the attribute category, and then based on the attribute category of the target sub-text, the target sub-text is transcribed into the target language to obtain a regularized sub-text with the same semantics as the target sub-text.

[0050] In one application scenario, the target language is Arabic, and there is a text to be regularized: Among them, "D", "3200", "1080", "×", "1920" are target sub-texts that need to be regularized because they are not in the standard form of Arabic. The remaining sub-texts are already in Arabic and do not need to be regularized. For "D", the aforementioned preset text rules can be used to determine the pronunciation of the English letter, and there is no need to use the attribute classification network for attribute category judgment and subsequent transcription. For the three numbers "3200", "1080", and "1920", it is necessary to use the attribute classification network to determine whether their numeral category belongs to a string, cardinality or ordinal number, as well as the specific gender and number category; and for the "×" multiplication sign, the attribute classification network also needs to determine the specific gender and number category. When training the attribute classification network, the target language is Arabic, and the regularized text is: The label "D" is the default label <self>, and the labels of "3200", "1080", and "1920" are the marks corresponding to the numeral category and the gender, number and case category, while the label of "×" is the mark corresponding to the gender, number and case category, so that <self>The subtext of the label will retain its original written form without conversion. For the labels of numbers and symbols, the original written form will be replaced by the regularized subtext with the same semantics as the target subtext.

[0051] Step S24: Detect the attribute categories of various attributes respectively to obtain detection results, including whether the attribute category of the attribute is correctly identified, and in response to the detection result including that the attribute category of the attribute is incorrectly identified, replace the attribute category of the corresponding attribute with a correction category.

[0052] In order to avoid interference between the classification results of target sub-texts with different attributes, for example, the symbol "×" multiplication sign should only have the gender and number category of the feminine genitive, but it gives a combination of the numeral category of the feminine genitive cardinal number and the gender and number category, therefore, after identifying the attribute categories of the target sub-text with respect to several attributes, the classification results are checked to filter out unreasonable classification results and improve the accuracy of the attribute categories. Specifically, after identifying the attribute categories of the target sub-text with respect to several attributes, and before transcribing the target sub-text into the target language based on the attribute categories of the target sub-text to obtain a regularized sub-text with the same semantics as the target sub-text, the attribute categories of various attributes are detected respectively to obtain a detection result; wherein, the detection result includes whether the attribute category of the attribute is correctly identified; in response to the detection result including that the attribute category of the attribute is incorrectly identified, the attribute category of the corresponding attribute is replaced with a correction category. Taking the "×" multiplication sign, which should originally have only the gender and number category of the feminine genitive, but gives a combination of the numeral category and the gender and number category of the cardinal number feminine genitive, as an example, the attribute categories of various attributes are detected to obtain detection results, and after it is found that the attribute category of the attribute is incorrectly identified, in response to the detection results including the incorrect identification of the attribute category of the attribute, the cardinal number feminine genitive is replaced with the correction category feminine genitive.

[0053] Step S24 may be performed as needed, that is, step S24 may not be performed in other disclosed embodiments.

[0054] Step S25: based on the attribute category of the target sub-text, transcribe the target sub-text into the target language to obtain a regularized sub-text having the same semantics as the target sub-text.

[0055] The description of step S25 can be found in Figure 1 Step S13 in the embodiment will not be described in detail here.

[0056] Step S26: Based on the regularized sub-text corresponding to the target sub-text, obtain the regularized text corresponding to the text to be regularized.

[0057] The description of step S26 can be found in Figure 1 Step S14 in the embodiment will not be repeated here. For the above-mentioned one public embodiment, after using the regular rule set of the target language to match the rules of several sub-texts respectively to obtain the target sub-text, and before detecting the attribute categories of various attributes respectively to obtain the detection results, the preset text rules of the target language can be used to match the preset text rules of several target sub-texts respectively, so that when the target sub-text that successfully matches the preset text rule is transcribed into a regularized sub-text expressed in the target language using the preset text rules, the regularized sub-text transcribed into the target language using the preset text rules and the attribute categories based on the target sub-text are summarized, and the target sub-text is transcribed into the target language to obtain a regularized sub-text with the same semantics as the target sub-text, and the regularized text corresponding to the text to be regularized is obtained.

[0058] In the above scheme, a regular rule set of the target language is used to perform rule matching on several sub-texts of the regular text to obtain the target sub-text, and several sub-texts that do not need regular processing are filtered out in advance to reduce the amount of data processing and improve the efficiency of text regularization; when identifying the attribute categories of the target sub-text regarding several attributes, the attribute categories of the several attributes are predicted by processing the regular text using an attribute classification network, which can improve the efficiency and accuracy of determining the attribute categories; by detecting the attribute categories of various attributes respectively, a detection result is obtained, the detection result includes whether the attribute category of the attribute is correctly identified, and in response to the detection result including that the attribute category of the attribute is incorrectly identified, the attribute category of the corresponding attribute is replaced with the correction category, which can avoid interference between the classification results of target sub-texts with different attributes.

[0059] See also Figure 3 , Figure 3 It is a schematic diagram of a framework of an embodiment of a text regularization device 30 of the present application. The text regularization device 30 includes a parsing module 31, a recognition module 32, a transcription module 33 and an acquisition module 34. The parsing module 31 is used to parse the text to be regularized to obtain a target subtext; wherein the text to be regularized is composed of a plurality of subtexts, the target subtext is the subtext to be regularized, the target subtext needs to be transcribed into a target language, and the grammar of the target language involves gender, number and case; the recognition module 32 is used to identify the attribute categories of the target subtext with respect to a plurality of attributes; wherein the plurality of attributes include gender, number and case attributes, and the attribute categories of the gender, number and case attributes include the gender, number and case categories of the target subtext in the target language; the transcription module 33 is used to transcribe the target subtext into the target language based on the attribute categories of the target subtext, and obtain a regularized subtext having the same semantics as the target subtext; the acquisition module 34 is used to obtain a regularized text corresponding to the text to be regularized based on the regularized subtext corresponding to the target subtext.

[0060] In the above scheme, the parsing module 31 parses the text to be regularized and obtains the target sub-text to be regularized. The identification module 32 identifies the attribute categories of the target sub-text regarding several attributes. The transcription module 33 transcribes the target sub-text into the target language based on the attribute categories of the target sub-text to obtain a regularized sub-text with the same semantics as the target sub-text. Finally, the acquisition module 34 obtains the regularized text corresponding to the text to be regularized based on the regularized sub-text corresponding to the target sub-text. Since this scheme first determines the attribute categories of the target sub-text and then uses the attribute categories to transcribe the text into the target language, classification is performed before transcription, thereby improving the accuracy and convenience of text regularization in numerical languages.

[0061] In some disclosed embodiments, when the target subtext is a number, the several attributes further include a numeral attribute, and the attribute category of the numeral attribute includes the numeral category of the target subtext.

[0062] Therefore, when the target sub-text is a number, the several attributes also include numeral attributes, and the attribute category of the numeral attribute includes the numeral category of the target sub-text, which can further enrich the attribute information of the target sub-text and improve the accuracy of subsequent transcription into the target language.

[0063] In some disclosed embodiments, when the target sub-text is a symbol, the several attributes further include symbol attributes, and the attribute category of the symbol attribute includes the semantic category of the target sub-text.

[0064] Therefore, when the target sub-text is a symbol, the several attributes also include symbol attributes, and the attribute categories of the symbol attributes include the semantic categories of the target sub-text, which can further enrich the attribute information of the target sub-text and improve the accuracy of subsequent transcription into the target language.

[0065] In some disclosed embodiments, the attribute categories of the several attributes are predicted by processing the text to be regularized using an attribute classification network, the attribute classification network is trained using sample text, and sample sub-texts in the sample text are annotated with sample tags; wherein, in the case where the sample sub-text needs to be regularized, the sample tag includes the sample attribute categories of the sample sub-text regarding the several attributes, and in the case where the sample sub-text does not need to be regularized, the sample tag is a preset tag.

[0066] Therefore, using the attribute classification network to assist in text regularization, taking attributes such as gender, number, and case, numeral attributes, and symbol attributes as a category and using the network for prediction, cleverly converting the transcription problem into a classification problem can improve the accuracy and convenience of text regularization for gender, number, and case languages. In addition, the training sample labels of the attribute classification network include preset labels and sample attribute categories for the several attributes, which not only distinguish whether regularization is needed, but also can process general language and gender, number, and case languages.

[0067] In some disclosed embodiments, the training step of the attribute classification network includes: obtaining a first vectorized embedding representation of the sample sub-text, obtaining N-gram character information of the sample sub-text, and obtaining a second vectorized embedding representation of the N-gram character information; fusing the first embedding representation and the second embedding representation to obtain a fused embedding representation of the sample sub-text; using the attribute classification network to perform classification prediction on the fused embedding representation to obtain a predicted label of the sample sub-text; wherein the predicted label is the preset label, or the predicted attribute category of the sample sub-text regarding the several attributes; and adjusting the network parameters of the attribute classification network based on the difference between the sample label and the predicted label.

[0068] Therefore, after fusing the first vectorized embedding representation of the sample sub-text and the second vectorized embedding representation of the N-gram character information, the attribute classification network will be used to perform classification prediction on the fused embedding representation to achieve the training of the attribute classification network. Based on this, adding appropriate auxiliary training information to the network input of the training attribute classification network can enhance the performance of the attribute classification network.

[0069] In some disclosed embodiments, the parsing module 31 is used to parse the text to be regularized and, when obtaining the target sub-text, is also used to analyze the text to be regularized to obtain the multiple sub-texts; using the regular rule set of the target language, the multiple sub-texts are respectively matched with the rules to obtain the target sub-texts; wherein the regular rule set includes at least one of a first subset and a second subset, the first subset includes multiple text rules that do not require regularization processing, and the second subset includes multiple text rules that require regularization processing.

[0070] Therefore, when parsing the text to be regularized and obtaining the target sub-text, the text to be regularized can be analyzed to obtain several sub-texts, and then the regular rule set of the target language can be used to match the rules of several sub-texts respectively to obtain the target sub-text. The sub-texts corresponding to several text rules that do not require regular processing can be filtered out in advance, and there is no need to perform subsequent classification and transcription, which reduces the amount of data processing and improves the efficiency of text regularization.

[0071] In some disclosed embodiments, the transcription module 33 is used to transcribe the target subtext into the target language based on the attribute category of the target subtext, and when obtaining a regularized subtext having the same semantics as the target subtext, it is also used to query in a transcription rule set to obtain a first subtext that satisfies a matching condition with the target subtext; wherein the transcription rule set includes a plurality of subtext pairs, the subtext pair includes a first subtext and a second subtext having the same semantics, and the second subtext is expressed in the target language according to the attribute category of the first subtext; and the second subtext in the subtext pair to which the first subtext belongs is used as the regularized subtext of the target subtext.

[0072] Therefore, when the second sub-text is expressed in the target language according to the attribute category of the first sub-text, a query is performed in the transcription rule set to obtain the first sub-text that meets the matching conditions with the target sub-text, and the second sub-text in the sub-text pair to which the first sub-text belongs can be used as the regularized sub-text of the target sub-text. Therefore, when determining the regularized sub-text based on the attribute category of the target sub-text, the transcription problem is cleverly converted into a classification problem, and classification is performed before transcription. This is suitable for scenarios where the same sub-text corresponds to multiple regularized sub-texts, and can solve the problem in the prior art that preset text rules and end-to-end models cannot perform regularized transcription of text according to the text context.

[0073] In some disclosed embodiments, the matching condition includes: the first subtext and the target subtext have the same semantics and the same attribute categories.

[0074] Therefore, through the matching conditions, the first sub-text with the same semantics and attribute categories as the target sub-text can be found, which is then used to subsequently determine the regularized sub-text, and the matching conditions are used to determine the unique correspondence to improve the transcription accuracy.

[0075] In some disclosed embodiments, after identifying the attribute categories of the target sub-text regarding several attributes, and before transcribing the target sub-text into the target language based on the attribute categories of the target sub-text to obtain a regularized sub-text having the same semantics as the target sub-text, the text regularization device 30 is further used to detect the attribute categories of various attributes respectively to obtain detection results; wherein the detection results include whether the attribute categories of the attributes are correctly identified; in response to the detection results including that the attribute categories of the attributes are incorrectly identified, replacing the attribute category corresponding to the attribute with a correction category.

[0076] Therefore, by detecting the attribute categories of various attributes, when the detection result includes incorrect identification of the attribute category of the attribute, the attribute category corresponding to the attribute is replaced with a corrected category, thereby avoiding interference in classification results between target sub-texts of different attributes.

[0077] In some disclosed embodiments, the acquisition module 34 is used to obtain the regularized text corresponding to the text to be regularized based on the regularized sub-text corresponding to the target sub-text, and is also used to replace the target sub-text in the text to be regularized with the regularized text corresponding to the target sub-text to obtain the regularized text.

[0078] Therefore, since the target subtext is the subtext that needs to be regularized in the text to be regularized, after obtaining the regularized subtext corresponding to the target subtext, the target subtext in the text to be regularized is replaced with the regularized text corresponding to the target subtext, so that the regularized text can be quickly obtained.

[0079] See also Figure 4 , Figure 4 1 is a schematic diagram of a framework of an embodiment of an electronic device 40 of the present application. The electronic device 40 includes a memory 41 and a processor 42 coupled to each other, the memory 41 stores program instructions, and the processor 42 is used to execute the program instructions to implement the steps in any of the above-mentioned text regularization method embodiments. Specifically, the electronic device 40 may include but is not limited to: a desktop computer, a laptop computer, a server, a mobile phone, a tablet computer, etc., which are not limited here.

[0080] Specifically, the processor 42 is used to control itself and the memory 41 to implement the steps in any of the above-mentioned text regularization method embodiments. The processor 42 can also be called a CPU (Central Processing Unit). The processor 42 may be an integrated circuit chip with signal processing capabilities. The processor 42 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 42 can be implemented by an integrated circuit chip.

[0081] The above scheme parses the text to be regularized, obtains the target sub-text to be regularized, identifies the attribute categories of the target sub-text regarding several attributes, and then transcribes the target sub-text into the target language based on the attribute categories of the target sub-text to obtain a regularized sub-text with the same semantics as the target sub-text, and finally obtains the regularized text corresponding to the text to be regularized based on the regularized sub-text corresponding to the target sub-text. Since this scheme first determines the attribute categories of the target sub-text and then uses the attribute categories to transcribe the text into the target language, it implements classification before transcription, thereby improving the accuracy and convenience of text regularization for numerical languages.

[0082] See also Figure 5 , Figure 5 1 is a schematic diagram of a framework of an embodiment of a computer-readable storage medium 50 of the present application. The computer-readable storage medium 50 stores program instructions 51 that can be executed by a processor, and the program instructions 51 are used to implement the steps in any of the above-mentioned text regularization method embodiments.

[0083] The above scheme parses the text to be regularized, obtains the target sub-text to be regularized, identifies the attribute categories of the target sub-text regarding several attributes, and then transcribes the target sub-text into the target language based on the attribute categories of the target sub-text to obtain a regularized sub-text with the same semantics as the target sub-text, and finally obtains the regularized text corresponding to the text to be regularized based on the regularized sub-text corresponding to the target sub-text. Since this scheme first determines the attribute categories of the target sub-text and then uses the attribute categories to transcribe the text into the target language, it implements classification before transcription, thereby improving the accuracy and convenience of text regularization for numerical languages.

[0084] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0085] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0086] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0087] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0088] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0089] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.< / self> < / self> < / self> < / self>

Claims

1. A text regularization method, characterized in that: include: Parsing the text to be regularized to obtain a target subtext; wherein the text to be regularized is composed of a plurality of subtexts, the target subtext is the subtext to be regularized, the target subtext needs to be transcribed into a target language, and the grammar of the target language involves gender, number and case; The target sub-texts are matched with the preset text rules by using the preset text rules of the target language, and the target sub-texts that successfully match the preset text rules are transcribed into regularized sub-texts represented in the target language by using the preset text rules, wherein the preset text rules are various transcription rules pre-set when performing text transcription; the target sub-texts that fail to match the preset text rules are input into an attribute classification network to identify the attribute categories of the target sub-texts with respect to several attributes; wherein the several attributes include gender, number, and case attributes, and the attribute categories of the gender, number, and case attributes include the following: the gender, number and case categories of the target subtext in the target language, the attribute classification network is obtained by training with sample texts, and the sample subtexts in the sample texts are annotated with sample tags, and when the sample subtexts need to be regularized, the sample tags include sample attribute categories of the sample subtexts with respect to the several attributes, and when the sample subtexts do not need to be regularized, the sample tags are preset tags; based on the attribute categories of the target subtext, the target subtext is transcribed into the target language to obtain a regularized subtext having the same semantics as the target subtext; Based on the regularized sub-text corresponding to the target sub-text, a regularized text corresponding to the text to be regularized is obtained.

2. The method according to claim 1, characterized in that In the case that the target subtext is a number, the plurality of attributes further include a numeral attribute, and the attribute category of the numeral attribute includes the numeral category of the target subtext.

3. The method according to claim 1 or 2, characterized in that: In the case that the target sub-text is a symbol, the several attributes further include symbol attributes, and the attribute categories of the symbol attributes include the semantic category of the target sub-text.

4. The method according to claim 1, characterized in that: The training steps of the attribute classification network include: Obtain a vectorized first embedding representation of the sample subtext, obtain N-gram character information of the sample subtext, and obtain a vectorized second embedding representation of the N-gram character information; fusing the first embedding representation and the second embedding representation to obtain a fused embedding representation of the sample subtext; Using the attribute classification network to perform classification prediction on the fused embedding representation, to obtain a predicted tag of the sample subtext; wherein the predicted tag is the preset tag, or a predicted attribute category of the sample subtext with respect to the several attributes; Based on the difference between the sample label and the predicted label, a network parameter of the attribute classification network is adjusted.

5. The method according to claim 1, characterized in that The parsing of the regularized text to obtain the target subtext includes: Analyzing the text to be regularized to obtain the plurality of sub-texts; Using the regular rule set of the target language, rule matching is performed on the plurality of subtexts respectively to obtain the target subtext; The regular rule set includes at least one of a first subset and a second subset, the first subset includes several text rules that do not require regular processing, and the second subset includes several text rules that require regular processing.

6. The method according to claim 1, characterized in that The step of transcribing the target subtext into the target language based on the attribute category of the target subtext to obtain a regularized subtext having the same semantics as the target subtext includes: Searching in a transcription rule set to obtain a first subtext that satisfies a matching condition with the target subtext; wherein the transcription rule set includes a plurality of subtext pairs, the subtext pairs include a first subtext and a second subtext having the same semantics, and the second subtext is expressed in the target language according to an attribute category of the first subtext; The second subtext in the subtext pair to which the first subtext belongs is used as the regularized subtext of the target subtext.

7. The method according to claim 6, characterized in that The matching condition includes: the first subtext and the target subtext have the same semantics and the same attribute categories.

8. The method according to claim 1, characterized in that: After identifying the attribute categories of the target subtext with respect to the plurality of attributes and before transcribing the target subtext into the target language based on the attribute categories of the target subtext to obtain a regularized subtext having the same semantics as the target subtext, the method further includes: Detecting the attribute categories of various attributes respectively to obtain detection results; wherein the detection results include whether the attribute categories of the attributes are correctly identified; In response to the detection result including that the attribute category of the attribute is incorrectly identified, the attribute category corresponding to the attribute is replaced with a corrected category.

9. The method according to claim 1, characterized in that: The step of obtaining the regularized text corresponding to the text to be regularized based on the regularized subtext corresponding to the target subtext includes: The target sub-text in the text to be regularized is replaced with the regularized text corresponding to the target sub-text to obtain the regularized text.

10. A text regularization device, characterized in that: include: A parsing module is used to parse the text to be regularized to obtain a target subtext; wherein the text to be regularized is composed of a plurality of subtexts, the target subtext is the subtext to be regularized, the target subtext needs to be transcribed into a target language, and the grammar of the target language involves gender, number and case; A recognition module is used to use the preset text rules of the target language to match the preset text rules for the target sub-texts respectively, and to transcribe the target sub-texts that successfully match the preset text rules into regularized sub-texts expressed in the target language using the preset text rules, wherein the preset text rules are various transcription rules preset when performing text transcription; the target sub-texts that fail to match the preset text rules are input into an attribute classification network to identify the attribute categories of the target sub-text with respect to several attributes; wherein the several attributes include gender, number, and case attributes, and the attribute categories of the gender, number, and case attributes include the gender, number, and case categories of the target sub-text in the target language, and the attribute classification network is obtained by training with sample texts, and the sample sub-texts in the sample texts are annotated with sample tags, and when the sample sub-texts need to be regularized, the sample tags include the sample attribute categories of the sample sub-texts with respect to the several attributes, and when the sample sub-texts do not need to be regularized, the sample tags are preset tags; A transcription module, configured to transcribe the target subtext into the target language based on the attribute category of the target subtext, so as to obtain a regularized subtext having the same semantics as the target subtext; The acquisition module is used to obtain the regularized text corresponding to the text to be regularized based on the regularized sub-text corresponding to the target sub-text.

11. An electronic device, characterized in that: The invention comprises a memory and a processor coupled to each other, wherein the memory stores program instructions, and the processor is used to execute the program instructions to implement the text regularization method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: Program instructions that can be executed by a processor are stored, and the program instructions are used to implement the text regularization method according to any one of claims 1 to 9.

Images