Text data labeling method, system and equipment, medium and product

Abstract

The invention is applicable to the technical field of data annotation, and provides a text data annotation method, system and device, a medium and a product, the method comprises the following steps: preprocessing input text data to obtain a to-be-annotated text; obtaining all to-be-matched texts, presetting scoring parameters, and calculating a lowest matching score based on the scoring parameters and the to-be-matched texts; based on a local sequence comparison algorithm and the scoring parameters, establishing a similarity calculation matrix between the to-be-labeled text and the to-be-matched text; based on the lowest matching score, an optimal matching path is obtained from the similarity calculation matrix, and the score of the optimal matching path in the similarity calculation matrix is the highest; and according to the optimal matching path, adding labeling information at a corresponding position of the input text. According to the method and the device, the user can adjust the scoring parameters according to own requirements, so that the annotation rule of the text data is flexibly customized, and efficient and accurate text annotation is provided.

Description

Technical Field

[0001] The present application belongs to the field of data annotation technology, and in particular, relates to a text data annotation method, system, device, medium and product. Background Art

[0002] Text data annotation is a key task in the field of artificial intelligence. The quality of data annotation directly affects the final operation effect of the artificial intelligence model. Text data annotation is widely used in scenarios such as information extraction, sentiment analysis, machine translation, and intelligent customer service. With the development of big data and artificial intelligence technology, automatic annotation technology has gradually become a key tool for processing and understanding massive text data.

[0003] In the prior art, a rule-based annotation method or a machine learning-based annotation method is usually used to annotate text data. For the rule-based annotation method, predefined rules and patterns are used to annotate the text, wherein the predefined rules are manually written by domain experts and contain specific keywords, phrase structures, context patterns, etc. This method relies on the rules manually written by domain experts and has high accuracy but poor scalability and generalization. For the machine learning-based annotation method, a neural network model (such as RNN, LSTM, Transformer, etc.) is used for text annotation. The specific method is to train the selected pre-trained model by using a large amount of annotated data and computing resources to enable the model to have the function of automatic annotation. However, this method requires a large amount of high-quality annotated data for training, the data acquisition cost is high, and the training data annotation process is time-consuming and labor-intensive.

[0004] In summary, the above two methods of text data annotation have poor generalization ability when dealing with new fields or specific text types, and cannot adapt to text data in different fields or contexts, which is more serious in some vertical fields with small amounts of data. At the same time, when dealing with large-scale text data, existing methods of text data annotation, especially deep learning models, are inefficient and may require a lot of computing resources in the inference stage, resulting in a long response time, making it difficult to meet the requirements of real-time processing. Summary of the invention

[0005] The embodiments of the present application provide a text data annotation method, system, device, medium and product, which can solve one of the above-mentioned problems in the prior art.

[0006] In a first aspect, an embodiment of the present application provides a text data annotation method, comprising:

[0007] Preprocess the input text data to obtain the text to be annotated;

[0008] Obtain all texts to be matched, preset scoring parameters, and calculate a minimum matching score based on the scoring parameters and the texts to be matched;

[0009] Based on the local sequence alignment algorithm and the scoring parameters, a similarity calculation matrix between the text to be annotated and the text to be matched is established;

[0010] Based on the minimum matching score, obtaining a best matching path from the similarity calculation matrix, wherein the best matching path has the highest score in the similarity calculation matrix;

[0011] According to the best matching path, annotation information is added at a corresponding position of the input text.

[0012] Further, the scoring parameters include a matching score, a mismatch penalty, a gap penalty and a matching degree threshold, wherein the matching score and the mismatch penalty are symmetrically arranged, and the gap penalty is lower than the mismatch penalty;

[0013] The calculation formula for calculating the minimum matching score is: minimum matching score = matching score × length of the text to be matched × matching degree threshold.

[0014] Furthermore, the similarity calculation matrix between the text to be annotated and the text to be matched is established based on the local sequence alignment algorithm and the scoring parameters, including:

[0015] Based on a preset word segmentation rule, word segmentation operations are performed on the text to be marked and the text to be matched respectively to obtain characters to be marked and characters to be matched;

[0016] Establishing an i×j primary matching matrix, initializing the values ​​of cells in the primary matching matrix to 0, wherein i=m+1, j=n+1, m is the number of characters to be matched, and n is the number of characters to be marked;

[0017] Based on the local sequence alignment algorithm, the value of each cell in the primary matching matrix is ​​iteratively calculated to obtain a similarity calculation matrix.

[0018] Furthermore, the calculation formula for iteratively calculating the value of each cell in the primary matching matrix is ​​as follows:

[0019]

[0020] Where matrix[i][j] represents the highest score of the cell in the i-th row and j-th column during the matching process. i and j are both integers, 1≤i≤m+1, 1≤j≤n+1, m is the number of characters to be matched, n is the number of characters to be marked, match is the matching score, mismatch is the mismatch penalty, and gap is the gap penalty.

[0021] Further, based on the minimum matching score, obtaining the best matching path from the similarity calculation matrix, wherein the best matching path has the highest score in the similarity calculation matrix, includes:

[0022] Filtering out all cells whose values ​​are greater than the minimum matching score from the similarity calculation matrix and marking them as matching cells;

[0023] Tracing back from the matching cell to the starting position to obtain multiple first matching paths, and marking the multiple first matching paths that partially overlap as second matching paths;

[0024] By comparing the values ​​of the matching cells of the plurality of second matching paths, the second matching path with the highest matching cell value is retained as the best matching path.

[0025] Furthermore, before adding annotation information to the corresponding position of the input text, the method includes:

[0026] A similarity calculation matrix is ​​established for each of the texts to be matched, and the best matching path of each of the texts to be matched is obtained from the texts to be annotated.

[0027] In a second aspect, an embodiment of the present application provides a text data annotation system, including:

[0028] Input text processing module: used to pre-process the input text data to obtain the text to be annotated;

[0029] Minimum matching score calculation module: used to obtain all the texts to be matched, preset scoring parameters, and calculate the minimum matching score based on the scoring parameters and the texts to be matched;

[0030] Similarity calculation matrix establishment module: used to establish a similarity calculation matrix between the text to be annotated and the text to be matched based on the local sequence alignment algorithm and the scoring parameters;

[0031] The best matching path obtaining module is used to obtain the best matching path from the similarity calculation matrix based on the minimum matching score, and the best matching path has the highest score in the similarity calculation matrix;

[0032] Annotation information adding module: used for adding annotation information at the corresponding position of the input text according to the best matching path.

[0033] In a third aspect, an embodiment of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned text data annotation method when executing the computer program.

[0034] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, including a computer-readable storage medium storing a computer program, and the computer program implements the above-mentioned text data annotation method when executed by a processor.

[0035] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed on a terminal device, enables the terminal device to execute the above-mentioned text data annotation method.

[0036] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0037] The present application discloses a text data annotation method, which adopts a local sequence alignment algorithm and can efficiently and accurately identify the best matching path in the input text, thereby improving the accuracy of annotation. Users can adjust the scoring parameters according to their own needs, and then flexibly customize the annotation rules of the text data. In addition, through the local sequence alignment algorithm, there is no need to rely on a large amount of training data and computing resources, which reduces data dependence and computing costs. It has obvious advantages in solving specific text matching problems in vertical fields, especially when dealing with smaller-scale data sets or scenarios with limited computing resources, and can provide efficient and accurate text annotation. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0039] Figure 1 It is a flowchart of a text data annotation method provided by one embodiment of the present invention;

[0040] Figure 2 is a schematic diagram of a primary matching matrix provided by an embodiment of the present invention;

[0041] Figure 3 is a schematic diagram of a similarity calculation matrix and an optimal matching path provided by an embodiment of the present invention;

[0042] Figure 4 It is a structural diagram of a text data annotation system provided by an embodiment of the present invention;

[0043] Figure 5 It is a structural diagram of the terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0045] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0046] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0047] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0048] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0049] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0050] See also Figure 1 As shown, the present invention is a text data annotation method, comprising the following steps:

[0051] S100, preprocessing the input text data to obtain the text to be annotated;

[0052] In this embodiment, the input text data needs to be preprocessed to remove rich text tags, special characters, paragraphs, sentences, etc., which can reduce noise and irrelevant information in the text, making the text content purer and standardized, thereby enhancing the readability and comprehensibility of the text, and helping subsequent processing steps to be more accurate and efficient.

[0053] S200, obtaining all texts to be matched, presetting scoring parameters, and calculating a minimum matching score based on the scoring parameters and the texts to be matched;

[0054] In this embodiment, by presetting the scoring parameters, the importance of the corresponding features or conditions in the matching process can be clearly defined, thereby establishing an objective and measurable matching standard, making the matching process more fair and transparent. In addition, when matching, by calculating the minimum matching score, the candidates with scores lower than the minimum matching score can be quickly excluded during the matching process, thereby narrowing the matching range and improving the matching efficiency.

[0055] In some embodiments, the scoring parameters include a match score, a mismatch penalty, a gap penalty, and a match threshold, wherein the match score and the mismatch penalty are symmetrically arranged, and the gap penalty is lower than the mismatch penalty;

[0056] The calculation formula for calculating the minimum matching score is: minimum matching score = matching score × length of the text to be matched × matching degree threshold.

[0057] In this embodiment, the matching score is the score obtained when the characters of the text to be annotated match the characters of the text to be matched, which is generally set to a positive value and is mainly used to give a positive match to the successfully matched characters; the mismatch penalty is the score obtained when the characters of the text to be annotated do not match the characters of the text to be matched, which is generally set to a negative value and is mainly used to give a reverse match to the incorrectly matched characters.

[0058] In this embodiment, in order to make the comparison result between the text to be annotated and the text to be matched more reasonable, a gap is introduced to indicate that there is an insertion / deletion operation between the text to be annotated and the text to be matched. In addition, in order to reflect the impact of the introduction of gaps between the text to be annotated and the text to be matched on the similarity, a gap penalty is set, which can reflect the score obtained when a gap is introduced in the matching process. It is generally set to a negative value, and is mainly used to give reverse feedback to the introduction of gaps.

[0059] In this embodiment, in order to ensure that there is sufficient similarity between the extracted text to be annotated and the text to be matched, a matching threshold is introduced. During the matching process, the user can filter out matching results that do not meet expectations by presetting the matching threshold, thereby simplifying the matching process.

[0060] In this embodiment, the scores of the match score and the mismatch penalty are symmetrically set, in order to make the importance of the match and the mismatch equal, and to ensure that there is a certain degree of difference between the text to be annotated and the text to be matched; and the score of the gap penalty is smaller than the score of the mismatch penalty, in order to allow a proper amount of insertion or deletion of characters in the matching process. In a preferred embodiment, the match score is 3 points, the mismatch penalty is -3 points, and the gap penalty is -2 points.

[0061] In this embodiment, since the score obtained when the characters of the text to be annotated and the characters of the text to be matched are completely matched is the theoretical highest similarity score, that is, the highest similarity score = the length of the text to be matched × the matching score, when calculating the minimum matching score, the expected minimum similarity between the text to be annotated and the text to be matched is calculated based on the theoretical highest similarity score, that is, the minimum matching score, wherein the minimum matching score = the highest similarity score × the matching threshold = the length of the text to be matched × the matching score × the matching threshold. In a preferred embodiment, the matching threshold is 0.7 points.

[0062] S300, establishing a similarity calculation matrix between the text to be annotated and the text to be matched based on the local sequence alignment algorithm and the scoring parameters;

[0063] In some embodiments, the step S300 includes:

[0064] Based on a preset word segmentation rule, word segmentation operations are performed on the text to be marked and the text to be matched respectively to obtain characters to be marked and characters to be matched;

[0065] Establishing an i×j primary matching matrix, initializing the values ​​of the cells of the primary matching matrix to 0, wherein i=m+1, j=n+1, m is the number of characters to be matched, and n is the number of characters to be marked;

[0066] Based on the local sequence alignment algorithm, the value of each cell in the primary matching matrix is ​​iteratively calculated to obtain a similarity calculation matrix.

[0067] In this embodiment, when establishing a primary matching matrix, the text to be marked and the text to be matched are respectively segmented using preset word segmentation rules to obtain characters to be marked and characters to be matched, wherein the word segmentation rules include a word segmentation method based on string matching, a word segmentation method based on statistics, a word segmentation method based on understanding, a word segmentation method based on rules, etc. For example, if a text to be matched is a string "TEST" and a text to be marked is a string "PTESATGAX", the text to be matched and the text to be marked are respectively segmented using a string-based word segmentation method to obtain characters T, E, S, T to be matched and characters P, T, E, S, A, T, G, A, X to be marked, wherein the length of the text to be matched is determined by the number of characters to be matched obtained after the word segmentation operation. Similarly, the length of the text to be marked is determined by the number of characters to be marked obtained after the word segmentation operation. Therefore, the length of the text to be matched is 4, and the length of the text to be marked is 9.

[0068] In this embodiment, taking the above-mentioned string to be matched "TEST" and the string to be annotated "PTESATGAX" as an example, the following is established: Figure 2 The primary matching matrix shown, specifically, in the primary matching matrix, according to the word segmentation results of the text to be matched "TEST" and the text to be annotated "PTESATGAX", an i×j primary matching matrix is ​​generated, wherein i=the number of characters to be matched + 1, j=the number of characters to be annotated + 1, and then the characters to be matched obtained after the word segmentation operation of the text to be matched are arranged in order from the second row. Similarly, the characters to be annotated obtained after the word segmentation operation of the text to be annotated are arranged in order starting from the second column, and at the same time, the value of each cell in the primary matching matrix is ​​initialized to 0 to obtain the above-mentioned primary matching matrix.

[0069] In some embodiments, the calculation formula for iteratively calculating the value of each cell in the primary matching matrix is ​​as follows:

[0070]

[0071] Where matrix[i][j] represents the highest score of the cell in the i-th row and j-th column during the matching process. i and j are both integers, 1≤i≤m+1, 1≤j≤n+1, m is the number of characters to be matched, n is the number of characters to be marked, match is the matching score, mismatch is the mismatch penalty, and gap is the gap penalty.

[0072] In this embodiment, based on the local sequence alignment algorithm, the values ​​of the cells in the primary matching matrix are filled to obtain a similarity calculation matrix, wherein the value of each cell in the similarity calculation matrix corresponds to the iterative similarity of the characters corresponding to the text to be matched and the text to be annotated.

[0073] In this embodiment, when the character to be matched and the character to be marked corresponding to the i-th row and j-th column are the same, the score of the corresponding cell is the matching score. Correspondingly, when the two characters corresponding to the i-th row and j-th column are different, the score of the corresponding cell is the mismatch penalty. Gaps are introduced in the above-mentioned text to be matched "TEST" and the text to be marked "PTESATGAX". When calculating the similarity scores of the corresponding cells, the gap penalties need to be considered.

[0074] In this embodiment, the value of each cell in the primary matching matrix is ​​determined by the maximum value of the iteration scores of the cells in three directions in the matrix, wherein the three directions include the upper left, above and left. The upper left is used to obtain the iteration matching score of the two characters corresponding to the current cell, the upper is used to obtain the iteration deletion score when the current cell has a deletion operation under the vacancy condition, and the left is used to obtain the iteration insertion score when the current cell has an iteration operation under the vacancy condition. In addition, when the iteration scores of the three directions of the cell are less than or equal to 0, the value of the cell is assigned to 0. Specifically, as shown in the calculation formula for the value of each cell in the above primary matching matrix, the iteration matching score in the matrix[i][j] cell is It is the cell value of matrix[i-1][j-1] cell + the match score or mismatch score of matrix[i][j] cell; the iterative deletion score in matrix[i][j] cell is the cell value of matrix[i-1][j] cell + the gap penalty; the iterative insertion score in matrix[i][j] cell is the cell value of matrix[i][j-1] cell + the gap penalty; at the same time, the assignment score in matrix[i][j] cell is 0. Therefore, by comparing the maximum values ​​of the iterative matching score, iterative deletion score, iterative insertion score and assignment score, the value of matrix[i][j] cell is determined.

[0075] In this embodiment, for matrix[0][j] and matrix[i][0], since there are no characters to be matched or characters to be marked, their cell values ​​are determined to be assigned a score of 0.

[0076] Specifically, based on the primary matching matrix in Table 1, with a matching score of 3 points, a mismatch penalty of -3 points, and a gap penalty of -2 points, we can obtain Figure 3 The similarity calculation matrix shown.

[0077] S400, based on the minimum matching score, obtaining a best matching path from the similarity calculation matrix, wherein the best matching path has the highest score in the similarity calculation matrix;

[0078] In some embodiments, the step S400 includes:

[0079] Filtering out all cells whose values ​​are greater than the minimum matching score from the similarity calculation matrix and marking them as matching cells;

[0080] Tracing back from the matching cell to a starting position to obtain a plurality of first matching paths, and marking the plurality of first matching paths that partially overlap as second matching paths;

[0081] By comparing the values ​​of the matching cells of the plurality of second matching paths, the second matching path with the highest matching cell value is retained as the best matching path.

[0082] In this embodiment, based on the lowest matching score, all matching cells in the similarity calculation matrix are screened out, and starting from the matching cell, each character to be marked in the text to be marked is traced back, and the cell with the highest value in the same column is marked as the backtracking cell, and the character to be marked corresponding to the backtracking cell is marked as the matching character. When the values ​​of the cells in the same column are all 0, the backtracking ends, and all matching characters are determined. Finally, starting from the matching cell, all matching characters are traced back to obtain the first matching path. Since there may be multiple matching cells in a similarity calculation matrix, there are multiple corresponding first matching paths in the similarity calculation matrix. The first matching paths that partially overlap in the multiple first matching paths are marked as second marking paths. By comparing the values ​​of the matching cells in the second matching path, it is helpful to quickly identify the best matching path in the similarity calculation matrix, and at the same time ensure the objectivity and accuracy of the selection of the best matching path.

[0083] In this embodiment, taking the matching degree threshold of 0.7 as an example, the calculation results are Figure 3The minimum matching score of the similarity calculation matrix shown is 3×4×0.7=8.4. All cells with values ​​greater than the minimum matching score are selected from the similarity calculation matrix, which are matrix[3][4]=9 and matrix[4][6]=10, and the above two cells are marked as matching cells. Starting from matrix[3][4]=9, each character to be marked in the text to be marked is backtracked to determine that the backtracked cells are matrix[2][3]=6 and matrix[1][2]=3, respectively. Thus, the first matching path S->E-> T. Similarly, starting from matrix[4][6]=10, backtrack the text to be marked to obtain the first matching path T->A->S->E->T. Thus, two first matching paths can be obtained from the above similarity calculation matrix. By comparing the above two first matching paths, it can be known that there is partial overlap in the two first matching paths. Therefore, the above two first matching paths are marked as second matching paths. In the above two second matching paths, the highest value of the matching cell is matrix[4][6]=10, so the best matching path can be determined as T->A->S->E->T.

[0084] S500: Add annotation information at a corresponding position of the input text according to the best matching path.

[0085] In some embodiments, before adding annotation information to the corresponding position of the input text, the method includes:

[0086] A similarity calculation matrix is ​​established for each of the texts to be matched, and the best matching path of each of the texts to be matched is obtained from the texts to be annotated.

[0087] In this embodiment, one input text corresponds to multiple texts to be matched, and a corresponding similarity calculation matrix is ​​established for each text to be matched. Through the above steps S200-S400, the best matching path of the similarity calculation matrix corresponding to the text to be matched is obtained, and all the best matching paths are marked as annotation information. When all the texts to be matched are matched, all the best matching paths are added to the corresponding positions of the input text to obtain the marking information of the text to be annotated.

[0088] See also Figure 4 As shown, the present invention also provides a text data annotation system, the system comprising:

[0089] Input text processing module 401: used to pre-process input text data to obtain text to be annotated;

[0090] Minimum matching score calculation module 402: used to obtain all the texts to be matched, preset scoring parameters, and calculate the minimum matching score based on the scoring parameters and the texts to be matched;

[0091] Similarity calculation matrix establishment module 403: used to establish a similarity calculation matrix between the text to be annotated and the text to be matched based on the local sequence alignment algorithm and the scoring parameters;

[0092] The best matching path obtaining module 404 is used to obtain the best matching path from the similarity calculation matrix based on the minimum matching score, wherein the best matching path has the highest score in the similarity calculation matrix;

[0093] The annotation information adding module 405 is used to add annotation information at the corresponding position of the input text according to the best matching path.

[0094] It is understandable that if Figure 1 The contents of the text data annotation method embodiment shown in the figure are applicable to the text data annotation system embodiment. The functions specifically implemented by the text data annotation system embodiment are similar to those in the figure. Figure 1 The text data annotation method embodiment shown is the same as that shown in the embodiment, and the beneficial effects achieved are the same as those shown in the embodiment. Figure 1 The beneficial effects achieved by the text data annotation method embodiment shown are also the same.

[0095] It should be noted that the information interaction, execution process and other contents between the above-mentioned systems are based on the same concept as the embodiment of the method of the present invention. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0096] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0097] See also Figure 5As shown, an embodiment of the present invention further provides a computer device 5, comprising: a memory 502 and a processor 501 and a computer program 503 stored in the memory 502. When the computer program 503 is executed on the processor 501, a text data annotation method as described in any one of the above methods is implemented.

[0098] The computer device 5 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The computer device 5 may include, but is not limited to, a processor 501 and a memory 502. Those skilled in the art will appreciate that Figure 5 It is only an example of the computer device 5 and does not constitute a limitation on the computer device 5. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0099] The processor 501 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0100] In some embodiments, the memory 502 may be an internal storage unit of the computer device 5, such as a hard disk or memory of the computer device 5. In other embodiments, the memory 502 may also be an external storage device of the computer device 5, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device 5. Further, the memory 502 may also include both an internal storage unit of the computer device 5 and an external storage device. The memory 502 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program, etc. The memory 502 may also be used to temporarily store data that has been output or is to be output.

[0101] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the text data annotation method as described in any one of the above methods is implemented.

[0102] In this embodiment, if the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0103] An embodiment of the present invention further provides a computer program product. When the computer program product runs on a mobile terminal, the mobile terminal can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0104] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A text data annotation method, characterized in that: include: Preprocess the input text data to obtain the text to be annotated; Obtain all texts to be matched, preset scoring parameters, and calculate a minimum matching score based on the scoring parameters and the texts to be matched; Based on the local sequence alignment algorithm and the scoring parameters, a similarity calculation matrix between the text to be annotated and the text to be matched is established; Based on the minimum matching score, obtaining a best matching path from the similarity calculation matrix, wherein the best matching path has the highest score in the similarity calculation matrix; According to the best matching path, annotation information is added at a corresponding position of the input text.

2. The method according to claim 1, characterized in that The scoring parameters include a matching score, a mismatch penalty, a gap penalty and a matching degree threshold, wherein the matching score and the mismatch penalty are symmetrically arranged, and the gap penalty is lower than the mismatch penalty; The calculation formula for calculating the minimum matching score is: minimum matching score = matching score × length of the text to be matched × matching degree threshold.

3. The method according to claim 2, characterized in that The method of establishing a similarity calculation matrix between the text to be annotated and the text to be matched based on the local sequence alignment algorithm and the scoring parameters includes: Based on a preset word segmentation rule, word segmentation operations are performed on the text to be marked and the text to be matched respectively to obtain characters to be marked and characters to be matched; Establishing an i×j primary matching matrix, initializing the values ​​of cells in the primary matching matrix to 0, wherein i=m+1, j=n+1, m is the number of characters to be matched, and n is the number of characters to be marked; Based on the local sequence alignment algorithm, the value of each cell in the primary matching matrix is ​​iteratively calculated to obtain a similarity calculation matrix.

4. The method according to claim 3, characterized in that The calculation formula for iteratively calculating the value of each cell in the primary matching matrix is ​​as follows: Where matrix[i][j] represents the highest score of the cell in the i-th row and j-th column during the matching process. i and j are both integers, 1≤i≤m+1, 1≤j≤n+1, m is the number of characters to be matched, n is the number of characters to be marked, match is the matching score, mismatch is the mismatch penalty, and gap is the gap penalty.

5. The method according to claim 1, characterized in that The obtaining, based on the minimum matching score, a best matching path from the similarity calculation matrix, wherein the best matching path has the highest score in the similarity calculation matrix, comprises: Filtering out all cells whose values ​​are greater than the minimum matching score from the similarity calculation matrix and marking them as matching cells; Tracing back from the matching cell to the starting position to obtain multiple first matching paths, and marking the multiple first matching paths that partially overlap as second matching paths; By comparing the values ​​of the matching cells of the plurality of second matching paths, the second matching path with the highest matching cell value is retained as the best matching path.

6. The method according to claim 1, characterized in that Before adding annotation information to the corresponding position of the input text, including: A similarity calculation matrix is ​​established for each of the texts to be matched, and the best matching path of each of the texts to be matched is obtained from the texts to be annotated.

7. A text data annotation system, characterized in that: include: Input text processing module: used to pre-process the input text data to obtain the text to be annotated; Minimum matching score calculation module: used to obtain all the texts to be matched, preset scoring parameters, and calculate the minimum matching score based on the scoring parameters and the texts to be matched; Similarity calculation matrix establishment module: used to establish a similarity calculation matrix between the text to be annotated and the text to be matched based on the local sequence alignment algorithm and the scoring parameters; The best matching path obtaining module is used to obtain the best matching path from the similarity calculation matrix based on the minimum matching score, and the best matching path has the highest score in the similarity calculation matrix; Annotation information adding module: used for adding annotation information at the corresponding position of the input text according to the best matching path.

8. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that: When the computer program product runs on a terminal device, the terminal device executes the method according to any one of claims 1 to 6.

Images