Text standardization processing method, device, electronic equipment and computer medium
By using pre-trained text component recognition model and standardized correction technology, address text of different writing methods is converted into a unified format, which solves the problem of irregular address text data in the prior art and improves the data statistics and management efficiency.
Patent Information
- Application Number
- CN202010773099.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-08-04
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2040-08-04
AI Technical Summary
The prior art is difficult to convert address text with different writing methods into a unified standardized format, resulting in inefficient address text data when stating and managing.
By obtaining the original text and splitting it into text components to be identified, the recognized text components are identified using the pre-trained text component recognition model, and the unidentified text components are standardized and corrected, and the text is finally converted into a unified standardized format.
It realizes the conversion of address texts with different writing methods into a unified format, improves the statistical and management efficiency of text data, and avoids the incomplete data statistics and inconvenience in management caused by irregular original text.
Smart Images

Figure CN111881680B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of data processing technology, and in particular to a text standardization processing method, a text standardization processing device, an electronic device, and a computer-readable medium. Background Art
[0002] In the process of collecting geographic location information, the collected address text data is usually not standardized and does not have a unified format and writing method, making it difficult to conduct further statistics and management.
[0003] In view of this, there is an urgent need in the art for a standardized processing method that can convert address texts written in different ways into texts in a unified standardized format.
[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention
[0005] The purpose of the present disclosure is to provide a text standardization processing method, a text standardization processing device, an electronic device and a computer-readable medium, thereby improving the efficiency of text data statistics and processing at least to a certain extent.
[0006] According to a first aspect of the present disclosure, a method for standardizing text is provided, comprising:
[0007] Acquire an original text to be processed, and split the original text to obtain various text components to be recognized in the original text;
[0008] Acquire a text component recognition model pre-trained by a standard text component set, and determine a first recognizable text component and an unrecognized text component in the text components to be recognized by using the text component recognition model;
[0009] Performing standardization correction on the unrecognized text component;
[0010] Determining a second recognizable text component in the corrected unrecognizable text component by using the text component recognition model; wherein the first recognizable text component and the second recognizable text component belong to the standard text component set;
[0011] A standardized text of the original text is obtained according to the first recognizable text component and the second recognizable text component.
[0012] In an exemplary embodiment of the present disclosure, the training process of the text component recognition model includes:
[0013] Acquire a standard text set, and split each standard text in the standard text set to obtain each standard text component in the standard text;
[0014] Adding corresponding type tags to the standard text components, and obtaining synonymous text components corresponding to each of the standard text components;
[0015] Obtaining the standard text component set according to the standard text components and the synonymous text components corresponding to each of the standard text components;
[0016] The text component recognition model is trained according to the standard text components with the type labels and corresponding synonymous text components in the standard text component set.
[0017] In an exemplary embodiment of the present disclosure, the determining, by the text component recognition model, the first recognizable text component and the unrecognized text component in the text component to be recognized includes:
[0018] After adding the corresponding type label to the text component to be identified, the label is input into the text component identification model;
[0019] Determine, by the text component recognition model, whether the text component to be recognized belongs to a standard text component or a synonymous text component corresponding to the type label in the standard text component set;
[0020] If the text component to be recognized belongs to the corresponding standard text component or synonymous text component in the type tag, the text component to be recognized is determined as a first recognizable text component;
[0021] If the text component to be recognized does not belong to the corresponding standard text component or synonymous text component in the type tag, the text component to be recognized is determined as an unrecognized text component.
[0022] In an exemplary embodiment of the present disclosure, after determining the first recognizable text component and the unrecognized text component in the text component to be recognized by the text component recognition model, the method further includes:
[0023] If the first recognizable text component is the synonymous text component, the first recognizable text component is converted into a standard text component corresponding to the synonymous text component.
[0024] In an exemplary embodiment of the present disclosure, the standardization correction of the unrecognized text component includes:
[0025] If the unrecognized text component is an erroneous text component, splitting the erroneous text component to obtain sub-text components of the erroneous text component;
[0026] Determining a correct text component corresponding to the erroneous text component from the set of standard text components according to the similarity between the sub-text component and the standard text component and the synonymous text component;
[0027] The erroneous text component is corrected according to the correct text component to obtain the corrected unrecognized text component.
[0028] In an exemplary embodiment of the present disclosure, the standardization correction of the unrecognized text component includes:
[0029] If the unrecognized text component is an abbreviated text component, matching the abbreviated text component with the standard text component and the synonymous text component, and determining a complete text component corresponding to the abbreviated text component from the set of standard text components;
[0030] The abbreviated text component is corrected according to the complete text component to obtain the corrected unrecognized text component.
[0031] In an exemplary embodiment of the present disclosure, obtaining a standardized text of the original text according to the first recognizable text component and the second recognizable text component includes:
[0032] Acquire application requirements of the original text, and acquire target text components from the first recognizable text components and the second recognizable text components according to the application requirements;
[0033] A standardized text of the original text is obtained according to the target text components in a preset text order.
[0034] According to a second aspect of the present disclosure, a text standardization processing device is provided, comprising:
[0035] The module for obtaining text to be recognized is used to obtain the original text to be processed and split the original text to obtain the various text components to be recognized in the original text;
[0036] A first text recognition module is used to obtain a text component recognition model pre-trained by a standard text component set, and determine a first recognizable text component and an unrecognized text component in the text component to be recognized by using the text component recognition model;
[0037] An unrecognized text correction module, used for performing standardization correction on the unrecognized text components;
[0038] A second text recognition module, configured to determine a second recognizable text component in the corrected unrecognizable text component by using the text component recognition model; wherein the first recognizable text component and the second recognizable text component belong to the standard text component set;
[0039] The standardized text determination module is used to obtain the standardized text of the original text according to the first recognizable text component and the second recognizable text component.
[0040] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-described methods for standardizing text by executing the executable instructions.
[0041] According to a fourth aspect of the present disclosure, a computer-readable medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the method for standardizing text described in any one of the above is implemented.
[0042] The exemplary embodiments of the present disclosure may have the following beneficial effects:
[0043] In the text standardization processing method of the example implementation mode of the present disclosure, each text component in the original text is identified by a pre-trained text component recognition model, and the unrecognized text components are corrected and recognized again. The original text containing different text components can be normalized into a standardized text with a unified format and writing method. When the above text is subsequently processed or used, the efficiency of text data statistics and management can be further improved, avoiding problems such as incomplete text data statistics and inconvenient management due to the non-standardization of the original text.
[0044] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.
[0046] Figure 1 A schematic diagram showing a flow chart of a method for standardizing text according to an exemplary embodiment of the present disclosure;
[0047] Figure 2A flowchart showing a training process of a text component recognition model according to an exemplary embodiment of the present disclosure is shown;
[0048] Figure 3 A schematic diagram of a process of identifying a text component to be identified by a text component identification model according to an exemplary embodiment of the present disclosure is shown;
[0049] Figure 4 A schematic diagram showing a process of performing standardized correction on erroneous text components according to an exemplary embodiment of the present disclosure;
[0050] Figure 5 A schematic diagram showing a process of performing standardization correction on abbreviation text components according to an exemplary embodiment of the present disclosure;
[0051] Figure 6 A block diagram showing a text standardization processing apparatus according to an exemplary embodiment of the present disclosure;
[0052] Figure 7 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0053] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.
[0054] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.
[0055] This example implementation first provides a method for standardizing text. Figure 1As shown, the above text standardization processing method may include the following steps:
[0056] Step S110: Obtain the original text to be processed, and split the original text to obtain the various text components to be recognized in the original text.
[0057] Step S120. Obtain a text component recognition model pre-trained by a standard text component set, and determine the first recognizable text component and unrecognized text components in the text components to be recognized by the text component recognition model.
[0058] Step S130: Standardize and correct the unrecognized text components.
[0059] Step S140. Determine a second recognizable text component in the corrected unrecognizable text component through a text component recognition model; wherein the first recognizable text component and the second recognizable text component belong to a standard text component set.
[0060] Step S150: Obtain a standardized text of the original text according to the first recognizable text component and the second recognizable text component.
[0061] The text standardization processing method in this example implementation can be applied to the standardization processing of various foreign language addresses, such as the standardization processing of address text data in multiple languages such as English and Malay.
[0062] Generally speaking, the collected address text data is usually not standardized. The same address may be written in many different ways, which makes it difficult to count and manage. Therefore, it is necessary to normalize these address texts and normalize these address texts with different writing methods into a unified standardized text. In this way, when conducting close contact tracing, the tracking data will not be incomplete due to the non-standard address.
[0063] For the standardization method of Chinese addresses, the address text can generally be standardized based on the domestic administrative division address standard table. For various Western addresses, the standardization processing method is different from the standardization method of Chinese addresses. On the one hand, Chinese texts are generally segmented before processing, while Western texts such as English need to undergo some preprocessing processes such as stemming before processing. On the other hand, in terms of the specific steps of the standardization method, there are some strategic differences between Western texts and Chinese texts. For example, when generating a probabilistic data model through a standard administrative division address text, some synonyms and conversions need to be combined with English writing habits; the processing of the original address text needs to be preprocessed through a normalized dictionary of some English words or phrase granularity; the processing of text components mixed with English and numbers, as well as abbreviation processing, have different strategic methods from the processing of Chinese text. In this example implementation, only the standardization processing of the original address information text in Malay and English is taken as an example to illustrate the specific implementation of the present disclosure. This method is also applicable to Western texts in other languages.
[0064] In the text standardization processing method of the example implementation mode of the present disclosure, each text component in the original text is identified by a pre-trained text component recognition model, and the unrecognized text components are corrected and recognized again. The original text containing different text components can be normalized into a standardized text with a unified format and writing method. When the above text is subsequently processed or used, the efficiency of text data statistics and management can be further improved, avoiding problems such as incomplete text data statistics and inconvenient management due to the non-standardization of the original text.
[0065] Next, combine Figures 2 to 5 The above steps of this exemplary embodiment are described in more detail.
[0066] In this example implementation, before the text standardization step is performed, the text component recognition model needs to be pre-trained using a standard text component set. If address text is to be processed, a standard address component set is obtained using a known standard terminology set for administrative division standard addresses, and then a data model for identifying standard address components is generated. Figure 2 As shown in the figure, the training process of the text component recognition model can specifically include the following steps:
[0067] Step S210: Obtain a standard text set, and split each standard text in the standard text set to obtain each standard text component in the standard text.
[0068] The standard text set refers to a set consisting of texts with standard formats and standard writing methods. The text component recognition model trained by the standard text set can recognize the text components with standard formats and writing methods in the original text.
[0069] If there is a known standard text set, you only need to perform some common text preprocessing, such as case conversion, punctuation processing, etc., and then proceed to the next step. If there is no standard text set, but only a relatively standard text set, in addition to the above text preprocessing, you also need to perform some steps such as component splitting to obtain a standard text set.
[0070] After obtaining the standard text set, the text data therein is split to obtain the corresponding standard text components. In English or Malay, the text data of the standard text set is generally split into word or phrase dimensions to obtain the corresponding standard text components.
[0071] Step S220. Add corresponding type tags to standard text components, and obtain synonymous text components corresponding to each standard text component.
[0072] In addition to the standard text components, it is also necessary to convert the standard text components into synonyms and synonym groups, establish some synonym conversion dictionaries, and obtain the synonym text components corresponding to each standard text component. In addition, add corresponding type labels to different standard text components. For example, the standard addresses of administrative divisions of a country mainly include: counties, districts, townships, streets, as well as island names and building names. The standard text components are marked according to these regional divisions.
[0073] Step S230. Obtain a standard text component set according to the standard text components and the synonymous text components corresponding to each standard text component.
[0074] All standard text components and their corresponding synonymous text components are put into a standard text component set as training data for model training.
[0075] Step S240: Train a text component recognition model based on the standard text components with type labels and corresponding synonymous text components in the standard text component set.
[0076] A data model that can be used to identify text components is established, such as a trie tree data model (i.e., a prefix tree data model, which is a data model used for text component identification methods), etc., and the model is trained using the above-mentioned training data with type labels to obtain a trained text component identification model for subsequent identification of original text components.
[0077] After training the text component recognition model through the above steps, the trained text component recognition model can be applied to this example implementation method. Figure 1 The method for standardizing text in is described in detail as follows:
[0078] In step S110, the original text to be processed is obtained, and the original text is split to obtain various text components to be recognized in the original text.
[0079] The raw text to be processed refers to text data directly obtained from various places with inconsistent formats, writing styles, etc., or with some spelling errors. Since such text data will cause great inconvenience in the subsequent statistics and management process, it needs to be standardized.
[0080] In this example implementation, the original text to be processed is first obtained, and the original text is split into word or phrase dimensions to obtain the various text components to be recognized in the original text. Then, some content in the text components to be recognized is converted through some pre-processing strategies, such as setting the type of letter numbers and other content after recognition, and converting word synonyms.
[0081] In step S120, a text component recognition model pre-trained by a standard text component set is obtained, and the first recognizable text component and unrecognized text components in the text components to be recognized are determined by the text component recognition model.
[0082] The pre-trained text component recognition model is obtained, and each text component to be recognized in the original text is recognized by the model to determine whether each text component to be recognized is in the standard text component set. The first recognizable text component belongs to the standard text component set, and the unrecognized text component does not belong to the standard text component set.
[0083] In this example implementation, Figure 3 As shown, determining the first identifiable text component and the unidentified text component in the text component to be identified by the text component identification model may specifically include the following steps:
[0084] Step S310: After adding the corresponding type label to the text component to be identified, input it into the text component identification model.
[0085] Before inputting each text component to be recognized in the original text into the text component recognition model, it is necessary to mark the type label of each text component to be recognized.
[0086] Step S320. Determine whether the text component to be identified belongs to a standard text component or a synonymous text component corresponding to a type label in a standard text component set through a text component identification model.
[0087] For example, if the text component to be recognized is a county name, the text component recognition model is used to determine whether a standard text component or a synonymous text component corresponding to the text component to be recognized can be matched in the type label of the county name.
[0088] Step S330: If the text component to be recognized belongs to the corresponding standard text component or synonymous text component in the type tag, the text component to be recognized is determined as the first recognizable text component.
[0089] If the corresponding standard text component or synonymous text component can be matched, it means that the text component to be identified belongs to the standard text component set and is determined as the first recognizable text component. The first recognizable text component refers to the text component to be identified that can be directly identified without error correction or standardization.
[0090] Step S340. If the text component to be recognized does not belong to the corresponding standard text component or synonymous text component in the type tag, the text component to be recognized is determined as an unrecognized text component.
[0091] If the corresponding standard text component or synonymous text component cannot be matched, it means that the text component to be recognized does not belong to the standard text component set, and it is determined as an unrecognized text component and needs to be processed and recognized again.
[0092] After determining the first recognizable text component and the unrecognized text component in the text component to be recognized by the text component recognition model, the method may further include:
[0093] If the first recognizable text component is a synonymous text component, the first recognizable text component is converted into a standard text component corresponding to the synonymous text component.
[0094] If the first identifiable text component is a synonymous text component, it needs to be converted into a unified standard text component to facilitate subsequent processing.
[0095] In step S130, the unrecognized text components are normalized and corrected.
[0096] Some unrecognized text components may be caused by spelling errors or abbreviations, so they need to be processed by the correction and standardization modules, and then recognized and processed by subsequent strategies again.
[0097] Unrecognized text components may include erroneous text components, abbreviated text components, and unnecessary text components. For erroneous text components and abbreviated text components, they can be standardized and corrected by some methods to be text components that can be recognized by the text component recognition model; and for unnecessary text components in the original text, they can be discarded after the recognition is completed.
[0098] For example, in the recognition of English address text components that are not in the standard address set, such as proper place names, block numbers, building numbers, floors, zip codes, etc., when identifying and processing them, the block numbers and building numbers have the same pattern, generally consisting of fixed words such as NO., UNIT, BLOCK, etc. followed by numbers. Although such address text components are not in the standard address set, they are also meaningful and need to be retained; some proper place names are also not in the standard address set. They may be aliases of a place or building names, or they are not included in the standard address set. Such address text components are also meaningful and need to be retained; in addition, the processing of some floors and zip codes is customized according to the composition characteristics of the address text. After identification, they can be used as components of the address, but they are generally not needed for subsequent use.
[0099] In this example implementation, Figure 4 As shown, standardization correction of erroneous text components may include the following steps:
[0100] Step S410. If the unrecognized text component is an erroneous text component, the erroneous text component is split to obtain sub-text components of the erroneous text component.
[0101] For the correction of the erroneous text components, the erroneous text components are first split into multiple granularities to obtain the sub-text components of the erroneous text components. For example, if an erroneous word is LUMAT, it is split into sub-text components of different granularities such as LUM, UMA, MAT, and LU.
[0102] Step S420. According to the similarity between the sub-text component and the standard text component and the synonymous text component, a correct text component corresponding to the erroneous text component is determined from the standard text component set.
[0103] An index relationship is established between each sub-text component obtained after the splitting and the standard text component and the synonymous text component, and the correct text component corresponding to the erroneous text component is determined from the standard text component set through recall filtering, scoring sorting, context verification and other steps.
[0104] Step S430. Correct the erroneous text component according to the correct text component to obtain a corrected unrecognized text component.
[0105] After the correct text components are determined, the erroneous text components are corrected according to the correct text components, and then the corrected erroneous text components are recognized again through the text component recognition model.
[0106] In this example implementation, Figure 5 As shown, standardization correction of abbreviation text components may include the following steps:
[0107] Step S510. If the unrecognized text component is an abbreviation text component, the abbreviation text component is matched with the standard text component and the synonymous text component, and a complete text component corresponding to the abbreviation text component is determined from the standard text component set.
[0108] For the correction of abbreviation text components, first match them with the standard text components and synonymous text components in the standard text component set to find standard text components or synonymous text components with common components, and then filter out the complete text components corresponding to the abbreviation text components through statistical and verification steps.
[0109] Step S520: Correct the abbreviated text component according to the complete text component to obtain a corrected unrecognized text component.
[0110] After the complete text components are determined, the abbreviated text components are corrected according to the complete text components, and the corrected abbreviated text components are recognized again through the text component recognition model.
[0111] In step S140, a second recognizable text component in the corrected unrecognized text component is determined by using a text component recognition model.
[0112] After correcting the erroneous text components and abbreviated text components in the unrecognized text components through the above steps, the corrected unrecognized text components are recognized again through the text component recognition model to find the second recognizable text component. The specific operation steps of the model recognition are the same as Figure 3 The steps in are exactly the same and will not be repeated here. The second recognizable text component is similar to the first recognizable text component and also belongs to the standard text component set, which may be a standard text component or a synonymous text component. If the second recognizable text component is a synonymous text component, it also needs to be converted into the corresponding standard text component.
[0113] In step S150, a standardized text of the original text is obtained according to the first recognizable text component and the second recognizable text component.
[0114] In this example implementation, the application requirements of the original text are obtained, and the target text component is obtained from the first recognizable text component and the second recognizable text component according to the application requirements, and then the standardized text of the original text is obtained according to the target text component in a preset text order.
[0115] For address text, the default text order is generally arranged in ascending order of place names. The structured data of each address text component outputted in the end includes standard address components as well as address components such as building number, floor, and proprietary place name. In subsequent applications, the required components can be selected according to different needs as the normalized result of the original address text.
[0116] For example, the original text of an address is "NO:8 SPG 286-30-123 RPN KAMPONG LUMUTBELAIT KC3135 BRUNEI DARUSSALAM". After being standardized through the above steps of this example implementation, the result is "NO. 8, SPG 286 - 30 - 123, RPN KAMPONG, KG LUMUT, BL", where "SPG286-30-123" is the street name, "KG LUMUT" is the village name, "BL" is the county name (BELAIT), "NO. 8" is the building number, and "RPN KAMPONG" is a proprietary place name, which is not included in the standard address set. In addition, "KC3135" is the postal code, and "BRUNEIDARUSSALAM" is the country name, which does not need to appear in the standardized address and therefore does not need to be retained.
[0117] It should be noted that although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.
[0118] Furthermore, the present disclosure also provides a text standardization processing device. Figure 6 As shown, the text standardization processing device may include a to-be-recognized text acquisition module 610, a first text recognition module 620, an unrecognized text correction module 630, a second text recognition module 640, and a standardized text determination module 650. Among them:
[0119] The to-be-recognized text acquisition module 610 can be used to acquire the original text to be processed and split the original text to obtain the various to-be-recognized text components in the original text;
[0120] The first text recognition module 620 may be used to obtain a text component recognition model pre-trained by a standard text component set, and determine a first recognizable text component and an unrecognized text component in the text component to be recognized by using the text component recognition model;
[0121] The unrecognized text correction module 630 may be used to perform standardization correction on the unrecognized text components;
[0122] The second text recognition module 640 may be used to determine a second recognizable text component in the corrected unrecognizable text component through a text component recognition model; wherein the first recognizable text component and the second recognizable text component belong to a standard text component set;
[0123] The standardized text determination module 650 may be configured to obtain a standardized text of the original text according to the first recognizable text component and the second recognizable text component.
[0124] In some exemplary embodiments of the present disclosure, a text standardization processing device provided by the present disclosure may further include a recognition model training module, which may include a standard text splitting unit, a synonymous text acquisition unit, a standard text component set determination unit, and a recognition model training unit. Wherein:
[0125] The standard text splitting unit can be used to obtain a standard text set, and split each standard text in the standard text set to obtain each standard text component in the standard text;
[0126] The synonymous text acquisition unit can be used to add corresponding type tags to standard text components and acquire synonymous text components corresponding to each standard text component;
[0127] The standard text component set determination unit may be used to obtain a standard text component set according to the standard text components and the synonymous text components corresponding to each standard text component;
[0128] The recognition model training unit can be used to train a text component recognition model based on standard text components with type labels and corresponding synonymous text components in a standard text component set.
[0129] In some exemplary embodiments of the present disclosure, the first text recognition module 620 may include a type tag adding unit, a text component judging unit, a first recognizable text component determining unit, and an unrecognized text component determining unit.
[0130] The type label adding unit can be used to add the corresponding type label to the text component to be identified and then input it into the text component identification model;
[0131] The text component judgment unit may be used to judge whether the text component to be recognized belongs to a standard text component or a synonymous text component corresponding to a type label in a standard text component set through a text component recognition model;
[0132] The first recognizable text component determining unit may be configured to determine the text component to be recognized as the first recognizable text component if the text component to be recognized belongs to a corresponding standard text component or a synonymous text component in the type tag;
[0133] The unrecognized text component determination unit may be configured to determine the text component to be recognized as an unrecognized text component if the text component to be recognized does not belong to a corresponding standard text component or a synonymous text component in the type tag.
[0134] In some exemplary embodiments of the present disclosure, the first text recognition module 620 may further include a synonymous text component conversion unit, which may be used to convert the first recognizable text component into a standard text component corresponding to the synonymous text component if the first recognizable text component is a synonymous text component.
[0135] In some exemplary embodiments of the present disclosure, the unrecognized text correction module 630 may include an erroneous text component separation unit, a correct text component determination unit, and an erroneous text component correction unit.
[0136] The error text component splitting unit may be used to split the error text component if the unrecognized text component is an error text component to obtain each sub-text component of the error text component;
[0137] The correct text component determination unit may be used to determine the correct text component corresponding to the erroneous text component from the standard text component set according to the similarity between the sub-text component and the standard text component and the synonymous text component;
[0138] The erroneous text component correction unit can be used to correct the erroneous text component according to the correct text component to obtain a corrected unrecognized text component.
[0139] In some exemplary embodiments of the present disclosure, the unrecognized text correction module 630 may further include a complete text component determination unit and an abbreviated text component correction unit.
[0140] The complete text component determination unit may be used to determine a complete text component corresponding to the abbreviated text component from a set of standard text components by matching the abbreviated text component with the standard text component and the synonymous text component if the unrecognized text component is an abbreviated text component;
[0141] The abbreviated text component correction unit may be used to correct the abbreviated text component according to the complete text component to obtain a corrected unrecognized text component.
[0142] In some exemplary embodiments of the present disclosure, the standardized text determination module 650 may include a target text component acquisition unit and a standardized text determination unit.
[0143] The target text component acquisition unit may be used to acquire application requirements of the original text, and acquire the target text component from the first recognizable text component and the second recognizable text component according to the application requirements;
[0144] The standardized text determination unit may be used to obtain a standardized text of the original text according to the target text components in a preset text order.
[0145] The specific details of each module / unit in the standardization processing device of the above text have been described in detail in the corresponding method embodiment part, and will not be repeated here.
[0146] Figure 7 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present invention is shown.
[0147] It should be noted that Figure 7 The computer system 700 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.
[0148] like Figure 7 As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 to a random access memory (RAM) 703. Various programs and data required for system operation are also stored in the RAM 703. The CPU 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.
[0149] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage section 708 as needed.
[0150] In particular, according to an embodiment of the present invention, the process described below with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 709, and / or installed from a removable medium 711. When the computer program is executed by the central processing unit (CPU) 701, various functions defined in the system of the present application are executed.
[0151] It should be noted that the computer-readable medium shown in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.
[0152] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0153] As another aspect, the present application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the method described in the following embodiment.
[0154] It should be noted that, although several modules of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules described above can be concretized in one module. Conversely, the features and functions of one module described above can be further divided into multiple modules to be concretized.
[0155] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modifications, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure.
[0156] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims
1. A method for standardizing text, characterized in that: include: Acquire an original text to be processed, and split the original text to obtain various text components to be recognized in the original text; Acquire a text component recognition model pre-trained by a standard text component set, and determine the first recognizable text component and the unrecognized text component in the text component to be recognized by the text component recognition model; wherein the standard text components in the standard text component set are obtained by splitting a standard text, and the standard text is a text with a standard format and a standard writing method; Performing standardization correction on the unrecognized text component; Determining a second recognizable text component in the corrected unrecognizable text component by using the text component recognition model; wherein the first recognizable text component and the second recognizable text component belong to the standard text component set; Obtaining a standardized text of the original text according to the first identifiable text component and the second identifiable text component; The step of performing standardization correction on the unrecognized text component includes: If the unrecognized text component is an erroneous text component, the erroneous text component is split to obtain sub-text components of the erroneous text component; according to the similarity between the sub-text component and the standard text component and the synonymous text component, a correct text component corresponding to the erroneous text component is determined from the standard text component set; the erroneous text component is corrected according to the correct text component to obtain the corrected unrecognized text component; If the unrecognized text component is an abbreviation text component, the abbreviation text component is matched with the standard text component and the synonymous text component, and a complete text component corresponding to the abbreviation text component is determined from the standard text component set; the abbreviation text component is corrected according to the complete text component to obtain the corrected unrecognized text component.
2. The text standardization processing method according to claim 1, characterized in that: The training process of the text component recognition model includes: Acquire a standard text set, and split each standard text in the standard text set to obtain each standard text component in the standard text; Adding corresponding type tags to the standard text components, and obtaining synonymous text components corresponding to each of the standard text components; Obtaining the standard text component set according to the standard text components and the synonymous text components corresponding to each of the standard text components; The text component recognition model is trained according to the standard text components with the type labels and corresponding synonymous text components in the standard text component set.
3. The text standardization processing method according to claim 1, characterized in that: The determining, by the text component recognition model, the first recognizable text component and the unrecognized text component in the text component to be recognized comprises: After adding the corresponding type label to the text component to be identified, the label is input into the text component identification model; Determine, by the text component recognition model, whether the text component to be recognized belongs to a standard text component or a synonymous text component corresponding to the type label in the standard text component set; If the text component to be recognized belongs to the corresponding standard text component or synonymous text component in the type tag, the text component to be recognized is determined as a first recognizable text component; If the text component to be recognized does not belong to the corresponding standard text component or synonymous text component in the type tag, the text component to be recognized is determined as an unrecognized text component.
4. The text standardization processing method according to claim 3 is characterized in that: After determining the first recognizable text component and the unrecognized text component in the text component to be recognized by the text component recognition model, the method further includes: If the first recognizable text component is the synonymous text component, the first recognizable text component is converted into a standard text component corresponding to the synonymous text component.
5. The text standardization processing method according to claim 1, characterized in that: The step of obtaining a standardized text of the original text according to the first identifiable text component and the second identifiable text component includes: Acquire application requirements of the original text, and acquire target text components from the first recognizable text components and the second recognizable text components according to the application requirements; A standardized text of the original text is obtained according to the target text components in a preset text order.
6. A text standardization processing device, characterized in that: include: The module for obtaining text to be recognized is used to obtain the original text to be processed and split the original text to obtain the various text components to be recognized in the original text; A first text recognition module is used to obtain a text component recognition model pre-trained by a standard text component set, and determine the first recognizable text component and the unrecognized text component in the text component to be recognized by the text component recognition model; wherein the standard text components in the standard text component set are obtained by splitting a standard text, and the standard text is a text with a standard format and a standard writing method; An unrecognized text correction module, used for performing standardization correction on the unrecognized text components; A second text recognition module, configured to determine a second recognizable text component in the corrected unrecognizable text component by using the text component recognition model; wherein the first recognizable text component and the second recognizable text component belong to the standard text component set; A standardized text determination module, configured to obtain a standardized text of the original text according to the first identifiable text component and the second identifiable text component; The step of performing standardization correction on the unrecognized text component includes: If the unrecognized text component is an erroneous text component, the erroneous text component is split to obtain sub-text components of the erroneous text component; according to the similarity between the sub-text component and the standard text component and the synonymous text component, a correct text component corresponding to the erroneous text component is determined from the standard text component set; the erroneous text component is corrected according to the correct text component to obtain the corrected unrecognized text component; If the unrecognized text component is an abbreviation text component, the abbreviation text component is matched with the standard text component and the synonymous text component, and a complete text component corresponding to the abbreviation text component is determined from the standard text component set; the abbreviation text component is corrected according to the complete text component to obtain the corrected unrecognized text component.
7. An electronic device, characterized in that: include: processor; as well as A memory for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the text standardization processing method as described in any one of claims 1 to 5.
8. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the text standardization processing method according to any one of claims 1 to 5 is implemented.
Citation Information
Patent Citations
Address standardization processing method, device, equipment and computer readable storage medium
CN110895651A
Text error correction method, device and system
CN111310440A
