A method, device and computer readable medium for data processing

Abstract

The embodiment of the present invention discloses a data processing method, device and computer-readable medium. The method first obtains a text pattern corresponding to a target area text; the target area includes at least two sub-areas, each of which has corresponding text; and determines a reference text pattern of the target area based on the text pattern; then determines whether the text of any sub-area conforms to the reference text pattern; if not, determines that the text of the sub-area is non-standard text. Thus, by determining the reference text pattern of the target area and using the reference text pattern to determine whether the text of the sub-area is non-standard text, the text content of the target area can be automatically detected, thereby improving the efficiency and accuracy of the target area text detection; and solves the time-consuming and labor-intensive problem of verifying the data content in the document by manual programming in the prior art.

Description

Technical Field

[0001] The present invention relates to the field of document editing technology, and in particular to a data processing method, device and computer-readable medium. Background Art

[0002] In the data table field, traditional spreadsheet tools (such as Excel, WPS office, etc.) can automatically generate filters for target columns or target rows; however, the filter result is only a simple classification and sorting of the same character strings in the target column or target row. In actual applications, we usually face more complex scenarios. For example, when the content of a cell in the target column is filled in incorrectly, it cannot be automatically identified and located by the above filter, and manual troubleshooting is required.

[0003] In order to determine whether the content of each cell in the target column or target row is correct, the current common practice in the industry is to program a column or a row to determine the judgment rule, and then make a judgment based on the judgment rule; however, this method has the following disadvantages: it requires extremely high professional skills from the user and requires programming; and when the spreadsheet data is huge, such as hundreds or thousands of columns of data, manually programming the rules will generate a huge workload, thereby affecting the document editing efficiency. Therefore, it is urgent to provide a data processing method to automatically check the data content in the data table and improve the document editing efficiency. Summary of the invention

[0004] In order to solve the problems existing in the above-mentioned prior art, the embodiments of the present invention provide a data processing method, device and computer-readable medium, which can automatically detect the text content of the target area in the document, improve the efficiency of document editing, and solve the time-consuming and labor-intensive problem caused by the use of manual programming to verify the data content in the document in the prior art.

[0005] To achieve the above-mentioned purpose, according to the first aspect of an embodiment of the present invention, a data processing method is provided, the method comprising: obtaining a text pattern corresponding to the text in a target area; the target area comprises at least two sub-areas, and the sub-areas have corresponding texts; determining a baseline text pattern of the target area based on the text pattern; determining whether the text of any sub-area conforms to the baseline text pattern; if not, determining that the text of the sub-area is non-standard text.

[0006] Optionally, determining a benchmark text pattern of a target area based on the text pattern includes: classifying and aggregating a number of the text patterns to obtain different types of text patterns; determining a weight corresponding to each type of text pattern; and determining a text pattern with the largest weight as a benchmark text pattern of the target area.

[0007] Optionally, obtaining the text pattern corresponding to the text in the target area includes: generating a regular expression based on the text in the sub-area; and traversing the text in all sub-areas in the target area to generate multiple regular expressions.

[0008] Optionally, the method further includes: determining that the non-standard text is correctable text; and correcting the correctable text based on the reference text pattern to obtain a corrected text.

[0009] Optionally, determining that the non-standard text is a correctable text includes: determining the similarity between a text pattern corresponding to the non-standard text and the benchmark text pattern; judging whether the similarity satisfies a preset condition; if the judgment result indicates that the similarity satisfies the preset condition, determining that the non-standard text is a correctable text.

[0010] Optionally, the determining that the non-standard text is a correctable text further includes: if the judgment result indicates that the similarity does not satisfy a preset condition, determining that the non-standard text is not a correctable text, and manually correcting the non-standard text.

[0011] Optionally, the correcting the correctable text based on the benchmark text pattern to obtain the corrected text includes: determining whether the correctable text contains preset characters; if the determination result indicates that the correctable text contains preset characters, then based on the benchmark text pattern, removing the preset characters from the correctable text to obtain the corrected text.

[0012] To achieve the above-mentioned purpose, according to the second aspect of an embodiment of the present invention, a data processing device is also provided, including: an acquisition module, used to acquire a text pattern corresponding to the text of a target area; the target area includes at least two sub-areas, each of which has corresponding text; a first determination module, used to determine a baseline text pattern of the target area based on the text pattern; a second determination module, used to determine whether the text of any sub-area conforms to the baseline text pattern; if not, determining that the text of the sub-area is non-standard text.

[0013] Optionally, the first determination module includes: an aggregation unit, used to classify and aggregate the plurality of text patterns to obtain different types of text patterns; a first determination unit, used to determine the weight corresponding to each type of text pattern; and a second determination unit, used to determine the text pattern with the largest weight as the benchmark text pattern for the target area.

[0014] To achieve the above objective, according to a third aspect of an embodiment of the present invention, there is further provided a computer-readable medium on which a computer program is stored, and when the program is executed by a processor, the data processing method as described in the first aspect is implemented.

[0015] Compared with the prior art, the data processing method, device and computer-readable medium provided by the embodiment of the present invention first obtain the text pattern corresponding to the text in the target area, the target area includes at least two sub-areas, each sub-area has corresponding text; secondly, the reference text pattern of the target area is determined based on the text pattern; then, it is determined whether the text of any sub-area conforms to the reference text pattern; if not, the text of the sub-area is determined to be non-standard text. Therefore, by determining the reference text pattern of the target area and using the reference text pattern to determine whether the text of the sub-area is non-standard text, the text content of the target area can be automatically detected, thereby improving the efficiency and accuracy of the target area text detection; and solving the time-consuming and labor-intensive problem of checking the data content in the document by manual programming in the prior art.

[0016] It should be understood that the teachings of the present invention are not required to achieve all of the beneficial effects described above, but specific technical solutions can achieve specific technical effects, and other embodiments of the present invention can also achieve beneficial effects not mentioned above. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings, the same or corresponding reference numerals represent the same or corresponding parts.

[0018] Figure 1 is a schematic flow chart of a data processing method according to an embodiment of the present invention;

[0019] Figure 2 is a schematic flow chart of a data processing method according to another embodiment of the present invention;

[0020] Figure 3 A schematic flow chart of a data processing method according to another embodiment of the present invention;

[0021] Figure 4 A schematic flow chart of determining a non-standard text as a correctable text in an embodiment of the present invention;

[0022] Figure 5 A schematic diagram of a data table and a data table processing process according to an embodiment of the present invention;

[0023] Figure 6 A schematic block diagram of a data processing device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0025] like Figure 1 As shown, a schematic flow chart of a data processing method according to an embodiment of the present invention. A data processing method, the method at least includes the following operation flow: S101, obtaining a text mode corresponding to the target area text; the target area includes at least two sub-areas, each sub-area has a corresponding text; S102, determining a reference text mode of the target area based on the text mode; S103, determining whether the text of any sub-area conforms to the reference text mode; if not, determining that the text of the sub-area is non-standard text.

[0026] In S101 , each sub-region has corresponding text, and each text has a corresponding text pattern.

[0027] Here, there is no limitation on the specific form and acquisition method of the text pattern. The text pattern can be a regular expression or other forms of expressions. For example, when the text pattern is a regular expression, the modified genetic algorithm can be used to calculate the regular expression corresponding to each sub-region text.

[0028] In S102, based on the text patterns corresponding to all sub-region texts, a reference text pattern of the target region is determined. Here, the reference text pattern is used to indicate a unified text pattern of all sub-regions in the target region. It can be that a text pattern that meets a preset condition is selected from a plurality of text patterns as the reference text pattern, or it can be that a plurality of text patterns are first aggregated, and then the aggregated text patterns are sorted according to weights, and finally the text pattern that is in the first place in the sorting result is determined as the reference text pattern.

[0029] In S103, the benchmark text pattern is used as a judgment rule; for the text of any sub-area: whether the text of the sub-area conforms to the benchmark text pattern is determined by judging whether the text matches the judgment rule; if the text matches the judgment rule, it is determined that the text of the sub-area conforms to the benchmark text pattern, and further determined that the text of the sub-area is standard text; if the text does not match the judgment rule, it is determined that the text of the sub-area does not conform to the benchmark text pattern, and further determined that the text of the sub-area is non-standard text.

[0030] This embodiment determines the reference text pattern of the target area based on the text patterns corresponding to all the texts in the target area; then for any sub-area: the reference text pattern is used to match the text of the sub-area, and if the match fails, the text of the sub-area is determined to be non-standard text. In this way, the text content of the target area can be automatically detected, avoiding the time-consuming and labor-intensive problems caused by manually programming judgment rules, and improving the efficiency and accuracy of target area detection.

[0031] It should be understood that in the embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0032] like Figure 2 As shown, a schematic flow chart of a data processing method according to another embodiment of the present invention. This embodiment is obtained by further optimizing the aforementioned embodiment. A data processing method, the method at least includes the following operation flow: S201, obtaining a text pattern corresponding to the text in the target area; the target area includes at least two sub-areas, each sub-area has a corresponding text; S202, classifying and aggregating a number of text patterns to obtain different types of text patterns; S203, determining the weight corresponding to each type of text pattern; S204, determining the text pattern with the largest weight as the benchmark text pattern of the target area; S205, determining whether the text in any sub-area conforms to the benchmark text pattern; if not, determining that the text in the sub-area is non-standard text.

[0033] Among them, the specific implementation process of S201 and S205 is the same as Figure 1 The specific implementation processes of S101 and S103 in the illustrated embodiment are similar and will not be described in detail here.

[0034] In S202 to S205, the types of text patterns and the number of sub-regions corresponding to each text pattern are counted; for any text pattern: based on the number of sub-regions corresponding to the text pattern and the number of all sub-regions in the target area, the weight corresponding to the text pattern is determined; the text patterns are sorted from large to small according to the weights, and the text pattern ranked first in the sorting result is determined as the benchmark text pattern of the target area.

[0035] This embodiment divides the text patterns of the target area into different types of text patterns by means of aggregation classification, calculates the weight corresponding to each text pattern, and then determines the text pattern with the largest weight as the baseline text pattern of the target area; thereby automatically determining the baseline text of the target area, which is facilitating the detection of the text content of the sub-area, and further realizing intelligent detection of the text content of the target area, thereby improving the accuracy and efficiency of the text content detection of the target area.

[0036] It should be understood that in the embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0037] like Figure 3 As shown, a schematic flow chart of a data processing method of another embodiment of the present invention. This embodiment is further optimized on the basis of the above-mentioned embodiment. A data processing method, the method at least includes the following operation flow: S301, obtaining the text pattern corresponding to the target area text; the target area includes at least two sub-areas, each sub-area has a corresponding text; S302, classifying and aggregating a number of text patterns to obtain different types of text patterns; S303, determining the weight corresponding to each type of text pattern; S304, determining the text pattern with the largest weight as the benchmark text pattern of the target area; determining the benchmark text pattern of the target area based on the text pattern; S305, determining whether the text of any sub-area conforms to the benchmark text pattern; if not, determining the text of the sub-area as non-standard text; S306, determining the non-standard text as correctable text; S307, correcting the correctable text based on the benchmark text pattern to obtain the corrected text.

[0038] Among them, the specific implementation process of S301, S302, S303, S304 and S305 is similar to the specific implementation process of S201, S202, S203, S204 and S205, and will not be repeated here.

[0039] In S306 and S307, the similarity between the text pattern corresponding to the non-standard text and the reference text pattern is calculated to determine whether the non-standard text is a correctable text. Alternatively, the non-standard text is determined to be a correctable text by determining whether the length of the non-standard text meets a preset length. For example: obtaining the text length of the non-standard text; determining whether the text length is greater than a preset threshold; if so, determining that the non-standard text is a correctable text; if not, determining that the non-standard text is not a correctable text, and manually correcting the non-standard text. Here, the preset threshold is used to indicate the length of the standard text.

[0040] Determine whether the correctable text contains preset characters; if the judgment result indicates that the correctable text contains preset characters, then based on the reference text mode, remove the preset characters from the correctable text to obtain the corrected text; if the judgment result indicates that the correctable text does not contain preset characters, then based on the reference text mode, perform content correction on the correctable text word by word to obtain the corrected text. For example, the text content in the target area is about a telephone number; the space in the telephone number is a preset character; determine whether the text mode corresponding to the correctable text contains the character symbol corresponding to the space, and if so, remove the space from the correctable text according to the reference text mode to obtain the corrected text.

[0041] This embodiment determines whether a non-standard text is a correctable text by whether the length of the non-standard text meets a preset length; then the correctable text is corrected based on the standard text pattern. Thus, it is possible to determine whether the non-standard text is a correctable text based on the baseline text pattern, and effectively correct the correctable text, thereby realizing automatic correction of the correctable text and improving the accuracy of text correction.

[0042] It should be understood that in the embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0043] like Figure 4 As shown, a schematic flow chart of determining a non-standard text as a correctable text in an embodiment of the present invention. This embodiment is further optimized on the basis of the above-mentioned embodiment. Determining a non-standard text as a correctable text, the method at least includes the following operation flow: S401, determining the similarity between the text pattern corresponding to the non-standard text and the reference text pattern; S402, determining whether the similarity meets the preset condition;

[0044] S403, if the judgment result indicates that the similarity meets the preset condition, the non-standard text is determined to be a correctable text;

[0045] S404: If the judgment result indicates that the similarity does not meet the preset condition, it is determined that the non-standard text is not a correctable text, and the non-standard text is manually corrected.

[0046] Manual correction is performed on non-standard texts, for example, manual correction is performed on non-standard texts based on standard text patterns.

[0047] Therefore, this embodiment can determine that the non-standard text is a correctable text based on the familiarity between the text pattern corresponding to the non-standard text and the reference text pattern, thereby improving the accuracy of correctable text judgment.

[0048] It should be understood that in the embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0049] The above embodiments are described in detail below in conjunction with application scenarios.

[0050] The target area C column in the data table mainly presents Emall; however, according to the content of column C, some cells are mistakenly filled with names, dates or project names. Existing tools are used to automatically identify the content of column C in the data table and generate filtering options. The filtering options only classify the contents of all cells in column C, but fail to identify and locate the wrong cells.

[0051] like Figure 5 As shown in FIG. 1 , a schematic diagram of a data table and a data table processing process according to an embodiment of the present invention is shown. For all cells in column C of the data table, the text pattern corresponding to the text in each cell is obtained, and several regular expressions are generated, such as Figure 5 a. Count the types of regular expressions and the number of cells corresponding to each regular expression; calculate the weight of each regular expression, and sort the regular expressions from large to small according to the weight, and determine the regular expression ranked first in the sorting result as the benchmark text pattern of column C, and then for any cell: use the benchmark text pattern as a judgment rule to determine whether the text in the cell conforms to the benchmark text pattern. If not, determine that the text in the cell is non-standard text; if it conforms, determine that the text in the cell is standard text; as shown in FIG. Figure 5 b; and so on, determine the cells corresponding to the non-standard text in column C; thereby, the text content in the cells can be automatically detected, thereby effectively obtaining the position information of the cells corresponding to the non-standard text.

[0052] In actual use, for wide data tables with hundreds or thousands of columns, it is impossible to manually program judgment rules for each column because the workload is huge; the method of this embodiment can be applied to big data processing, which can effectively improve the efficiency and accuracy of document editing.

[0053] like Figure 6As shown, it is a schematic block diagram of a data processing device according to an embodiment of the present invention. A data processing device, the device 600 comprises: an acquisition module 601, used to acquire a text pattern corresponding to a target area text; the target area comprises at least two sub-areas, each of which has corresponding text; a first determination module 602, used to determine a reference text pattern of the target area based on the text pattern; a second determination module 603, used to determine whether the text of any sub-area conforms to the reference text pattern; if not, the text of the sub-area is determined to be a non-standard text.

[0054] In an optional embodiment, the first determination module 602 includes: an aggregation unit, used to classify and aggregate the plurality of text patterns to obtain different types of text patterns; a first determination unit, used to determine the weight corresponding to each type of text pattern; and a second determination unit, used to determine the text pattern with the largest weight as the benchmark text pattern for the target area.

[0055] In an optional embodiment, the acquisition module 601 includes: a first generation unit, which is used to generate a regular expression based on the text of the sub-region; and a second generation unit, which is used to traverse the text of all sub-regions in the target region to generate multiple regular expressions.

[0056] In an optional embodiment, the device further includes: a third determination module, used to determine that the non-standard text is a correctable text; and a correction module, used to correct the correctable text based on the reference text pattern to obtain a corrected text.

[0057] In an optional embodiment, the third determination module includes: a first determination unit, used to determine the similarity between the text pattern corresponding to the non-standard text and the benchmark text pattern; a judgment unit, used to judge whether the similarity meets a preset condition; and a second determination unit, used to determine that the non-standard text is a correctable text if the judgment result indicates that the similarity meets the preset condition.

[0058] In an optional embodiment, the third determination module further includes: a third determination unit, configured to determine that the non-standard text is not a correctable text and manually correct the non-standard text if the judgment result indicates that the similarity does not satisfy a preset condition.

[0059] In an optional embodiment, the correction module includes: a judgment unit, used to judge whether the correctable text contains preset characters; a correction unit, used to remove the preset characters from the correctable text based on the reference text pattern to obtain corrected text if the judgment result indicates that the correctable text contains preset characters.

[0060] The above device can execute the data processing method provided by an embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the data processing method. For technical details not described in detail in this embodiment, please refer to the data processing method provided by the embodiment of the present invention.

[0061] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0062] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another device, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0063] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0064] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0065] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage medium, read-only memory (ROM), disk or optical disk, etc. Various media that can store program codes.

[0066] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods of each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage media, ROM, magnetic disks or optical disks.

[0067] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A data processing method, comprising: Acquire a text mode corresponding to a target area text in a data table; the target area includes at least two sub-areas, each of which has corresponding text; determining a reference text pattern for a target area based on the text pattern; Determining whether the text of any sub-region conforms to the reference text pattern; If not, the text in the sub-region is determined to be non-standard text; The determining a reference text pattern of a target area based on the text pattern comprises: Classify and aggregate all the text patterns to obtain different types of text patterns; Determine the weight corresponding to each type of text pattern; The text pattern with the largest weight is determined as the benchmark text pattern of the target area.

2. According to the method of claim 1, the step of obtaining the text mode corresponding to the text in the target area comprises: Generate a regular expression based on the text in the sub-region; Traverse the text of all sub-regions in the target region and generate multiple regular expressions.

3. The method according to claim 1, further comprising: Determining that the non-standard text is a correctable text; The correctable text is corrected based on the reference text pattern to obtain a corrected text.

4. The method according to claim 3, wherein determining that the non-standard text is a correctable text comprises: Determining the similarity between the text pattern corresponding to the non-standard text and the reference text pattern; Determining whether the similarity satisfies a preset condition; If the judgment result indicates that the similarity satisfies a preset condition, the non-standard text is determined to be a correctable text.

5. The method according to claim 4, further comprising: If the judgment result indicates that the similarity does not meet the preset condition, it is determined that the non-standard text is not a correctable text, and the non-standard text is manually corrected.

6. The method according to claim 3, wherein the correctable text is corrected based on the reference text pattern to obtain a corrected text, comprising: Determining whether the correctable text contains preset characters; If the judgment result indicates that the correctable text contains preset characters, then based on the reference text pattern, the preset characters are removed from the correctable text to obtain a corrected text.

7. A data processing device, comprising: An acquisition module is used to acquire the text mode corresponding to the text in the target area of ​​the data table; The target area includes at least two sub-areas, each of which has corresponding text; A first determination module, configured to determine a reference text pattern of a target area based on the text pattern; A second determination module, used to determine whether the text of any sub-region conforms to the reference text pattern; If not, the text in the sub-region is determined to be non-standard text; The first determining module comprises: An aggregation unit, used for classifying and aggregating all the text patterns to obtain different types of text patterns; A first determining unit, used to determine the weight corresponding to each type of text mode; The second determining unit is used to determine the text pattern with the largest weight as the reference text pattern of the target area.

8. A computer readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images