Text representation method and device, storage medium and electronic equipment

Abstract

The invention provides a text representation method and device, a storage medium and electronic equipment, and the method comprises the steps: obtaining an input statement text; performing word segmentation on the statement text to obtain a plurality of word segmentation data; each piece of word segmentation data is vectorized, a word segmentation matrix of each piece of word segmentation data is obtained, the word segmentation matrix comprises a word root vector, a part-of-speech vector and a lexeme vector, and the part-of-speech vector is used for representing part-of-speech information of the word segmentation data; and splicing the word segmentation matrixes of the plurality of word segmentation data to obtain a target statement matrix of the statement text. By constructing the target statement matrix comprising the root vector, the part-of-speech vector and the lexeme vector, richness and accuracy of text representation are enhanced. When the target statement matrix is used for training the language processing model, the language processing model can accurately learn the part-of-speech information of the Chinese text, and the learning accuracy and the understanding ability of the language processing model are improved.

Description

Technical Field

[0001] The present disclosure relates to the field of natural language processing, and in particular to a text representation method, device, storage medium and electronic device. Background Art

[0002] With the rapid development of information technology, NLP (Natural Language Processing) technology is increasingly used in various fields. Text representation is the basis of NLP tasks. Text representation refers to the process of converting text data into digital form for calculation and analysis. Through text representation, we can map natural language text into a mathematical space so that machine learning algorithms can process this data. Therefore, text representation directly affects the performance and accuracy of machine learning models.

[0003] At present, text representation methods are mainly based on deep learning text representation methods, that is, through deep learning algorithms such as word embedding technology, the semantic similarity and contextual information between words are captured. However, current text representation methods have certain limitations, such as the inability to effectively distinguish polysemy and the neglect of the importance of part of speech for semantic understanding. Summary of the invention

[0004] In view of this, the present disclosure provides a text representation method, device, storage medium and electronic device, which can improve the richness and accuracy of text representation.

[0005] According to a first aspect of an embodiment of the present disclosure, a text representation method is provided, the method comprising:

[0006] Get the input statement text;

[0007] Segmenting the sentence text to obtain a plurality of segmentation data;

[0008] Vectorize each word segmentation data to obtain a word segmentation matrix for each word segmentation data, wherein the word segmentation matrix includes a root vector, a part-of-speech vector and a word position vector, and the part-of-speech vector is used to represent the part-of-speech information of the word segmentation data;

[0009] The word segmentation matrices of the plurality of word segmentation data are concatenated to obtain a target sentence matrix of the sentence text.

[0010] In one embodiment, the sentence text is segmented to obtain a plurality of segmentation data, including:

[0011] Obtaining a predefined word segmentation dictionary, wherein the word segmentation dictionary includes a plurality of known words;

[0012] Matching the sentence text with the word segmentation dictionary to obtain a sentence matching result;

[0013] The sentence text is segmented according to the sentence matching result to obtain the multiple segmentation data.

[0014] In one embodiment, vectorizing each word segmentation data to obtain a word segmentation matrix for each word segmentation data includes:

[0015] Identify the root information of each word segmentation data and generate the root vector of each word segmentation data;

[0016] Identify the part-of-speech information of each word segmentation data and generate a part-of-speech vector for each word segmentation data;

[0017] Identify the word position information of each word segmentation data and generate the word position vector of each word segmentation data;

[0018] The root vector, part-of-speech vector and word position vector of each word segmentation data are concatenated to obtain the word segmentation matrix of each word segmentation data.

[0019] In one embodiment, identifying the part-of-speech information of each word segmentation data and generating a part-of-speech vector for each word segmentation data includes:

[0020] Obtaining a preset part-of-speech dictionary, wherein the part-of-speech dictionary includes part-of-speech classification information of a plurality of word segmentation data;

[0021] According to the part-of-speech dictionary, the part-of-speech information of each word segmentation data is identified and the part-of-speech vector is generated.

[0022] In one embodiment, the step of concatenating the segmentation matrices of the plurality of segmentation data to obtain a target sentence matrix of the sentence text includes:

[0023] Determining the sequence information of the plurality of word segmentation data according to the word position vector;

[0024] The word segmentation matrices of the plurality of word segmentation data are arranged according to the sequence information to obtain the target sentence matrix.

[0025] In one embodiment, the word segmentation matrices of the plurality of word segmentation data are arranged according to the sequence information to obtain a target sentence matrix of the sentence text, including:

[0026] Arranging the word segmentation matrix according to the sequence information to obtain an initial sentence matrix, wherein the initial sentence matrix includes a root column vector, a part-of-speech column vector, and a word position column vector;

[0027] The coefficients of the root column vector, the part-of-speech column vector and the word position column vector in the initial sentence matrix are set to obtain the target sentence matrix.

[0028] According to a second aspect of an embodiment of the present disclosure, a text representation device is provided, the device comprising:

[0029] An acquisition unit, used for acquiring an input sentence text;

[0030] A word segmentation unit, used for segmenting the sentence text to obtain a plurality of word segmentation data;

[0031] A processing unit, used to vectorize each word segmentation data to obtain a word segmentation matrix of each word segmentation data, wherein the word segmentation matrix includes a root vector, a part-of-speech vector and a word position vector, and the part-of-speech vector is used to represent the part-of-speech information of the word segmentation data;

[0032] The concatenation unit is used to concatenate the word segmentation matrices of the plurality of word segmentation data to obtain a target sentence matrix of the sentence text.

[0033] In one embodiment, the word segmentation unit is used to:

[0034] Obtaining a predefined word segmentation dictionary, wherein the word segmentation dictionary includes a plurality of known words;

[0035] Matching the sentence text with the word segmentation dictionary to obtain a sentence matching result;

[0036] The sentence text is segmented according to the sentence matching result to obtain the multiple segmentation data.

[0037] In one embodiment, the processing unit is used to:

[0038] Identify the root information of each word segmentation data and generate the root vector of each word segmentation data;

[0039] Identify the part-of-speech information of each word segmentation data and generate a part-of-speech vector for each word segmentation data;

[0040] Identify the word position information of each word segmentation data and generate the word position vector of each word segmentation data;

[0041] The root vector, part-of-speech vector and word position vector of each word segmentation data are concatenated to obtain the word segmentation matrix of each word segmentation data.

[0042] In one embodiment, the processing unit is used to:

[0043] Obtaining a preset part-of-speech dictionary, wherein the part-of-speech dictionary includes part-of-speech classification information of a plurality of word segmentation data;

[0044] According to the part-of-speech dictionary, the part-of-speech information of each word segmentation data is identified and the part-of-speech vector is generated.

[0045] In one embodiment, the splicing unit is used to:

[0046] Determining the sequence information of the plurality of word segmentation data according to the word position vector;

[0047] The word segmentation matrices of the plurality of word segmentation data are arranged according to the sequence information to obtain the target sentence matrix.

[0048] In one embodiment, the splicing unit is used to:

[0049] Arranging the word segmentation matrix according to the sequence information to obtain an initial sentence matrix, wherein the initial sentence matrix includes a root column vector, a part-of-speech column vector, and a word position column vector;

[0050] The coefficients of the root column vector, the part-of-speech column vector and the word position column vector in the initial sentence matrix are set to obtain the target sentence matrix.

[0051] According to a third aspect of an embodiment of the present disclosure, there is provided a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of any method described in the first aspect are implemented.

[0052] According to a fourth aspect of an embodiment of the present disclosure, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any method described in the first aspect are implemented.

[0053] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, which, when executed by a processor, implements

[0054] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0055] By constructing a target sentence matrix including root vectors, part-of-speech vectors, and word position vectors, the richness and accuracy of text representation are enhanced. When using the target sentence matrix to train a language processing model, the language processing model can accurately learn the part-of-speech information of Chinese text, improving the learning accuracy and comprehension ability of the language processing model. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 is a flow chart of a text representation method shown in an exemplary embodiment of the present disclosure;

[0057] Figure 2 is a flow chart of a method for word segmentation calculation shown in an exemplary embodiment of the present disclosure;

[0058] Figure 3 is a flow chart of a word segmentation data vectorization method shown in an exemplary embodiment of the present disclosure;

[0059] Figure 4 is a flow chart of a method for calculating part-of-speech vectorization shown in an exemplary embodiment of the present disclosure;

[0060] Figure 5 is a flow chart of a word segmentation matrix concatenation method shown in an exemplary embodiment of the present disclosure;

[0061] Figure 6 is a flow chart of a target sentence matrix concatenation method shown in an exemplary embodiment of the present disclosure;

[0062] Figure 7 is a flow chart of another text representation method shown in an exemplary embodiment of the present disclosure;

[0063] Figure 8 is a block diagram of a text representation device shown in an exemplary embodiment of the present disclosure;

[0064] Fig. 9 It is a hardware structure diagram of an electronic device shown in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0065] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0066] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. The singular forms of "a", "said" and "the" used in this disclosure and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0067] It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0068] Text representation is the basis of NLP tasks. It represents text in a format that can be processed by computers to facilitate subsequent training of language processing models. Language processing models can be applied to a variety of scenarios, such as search engine optimization, machine translation, intelligent customer service, sentiment analysis, etc. Accurate and efficient text representation algorithms can improve the understanding of text data by subsequent language processing models, thereby improving the accuracy of NLP tasks.

[0069] At present, text representation methods are mainly based on deep learning text representation methods, that is, through deep learning algorithms such as word embedding technology, the semantic similarity and contextual information between words are captured. Specifically, the semantic similarity and contextual information between words can be captured by calculating the root vector and word position vector of the word. However, for Chinese text, there are many polysemy phenomena in Chinese text. For the same word, the meaning in different sentences may be completely different.

[0070] Based on this, the present disclosure provides a text representation method, which is applied to a computer device to improve the richness and accuracy of text representation. For details, please refer to Figure 1 The flowchart shown may include the following steps:

[0071] S101, obtaining input sentence text.

[0072] In some embodiments, sentence text for training a language processing model is first obtained.

[0073] S102, segmenting the sentence text to obtain a plurality of segmentation data.

[0074] In some embodiments, the most basic unit of the language processing model to understand text is words. Therefore, after obtaining the sentence text, it is necessary to first perform word segmentation calculation on the sentence text. Figure 2 The flowchart shown may include the following steps:

[0075] S201, obtaining a predefined word segmentation dictionary.

[0076] S202, matching the sentence text with the word segmentation dictionary to obtain a sentence matching result.

[0077] S203, segmenting the sentence text according to the sentence matching result to obtain the plurality of segmentation data.

[0078] When performing word segmentation on a sentence text, it is first necessary to obtain a predefined word segmentation dictionary, which includes multiple known words and word segmentation rules. Then the sentence text is matched with the known words in the word segmentation dictionary to achieve preliminary word segmentation of the sentence text and obtain a sentence matching result. The sentence matching result is the vocabulary information existing in the dictionary. Furthermore, according to the word segmentation rules, the boundaries of the sentence matching result are processed, for example, the sentence matching result is processed into a compound word to obtain the final word segmentation vocabulary.

[0079] In addition to segmenting the text of a sentence based on a segmentation dictionary, you can also use an open source segmentation tool to segment the text of a sentence, for example, using Jieba segmentation. Furthermore, you can also use machine learning or deep learning methods to segment the text of a sentence. The specific segmentation method can be selected according to the amount of data in the sentence text and the segmentation requirements.

[0080] By segmenting the sentence text, the sentence text is converted into segmented data that is easy for the language processing model to understand, which is beneficial to improving the comprehension ability of the language processing model. At the same time, the sentence text is vectorized in units of vocabulary, which improves the accuracy of the sentence text vectorization.

[0081] S103, vectorizing each word segmentation data to obtain a word segmentation matrix for each word segmentation data.

[0082] After the sentence text is segmented and multiple segmentation data in the sentence are obtained, each segmentation data needs to be vectorized and represented by digital information. When the segmentation data is vectorized, methods such as dictionary matching and deep learning models can be used. After the segmentation data is vectorized, the segmentation matrix of each segmentation data can be obtained. The segmentation matrix includes root vectors, part-of-speech vectors, and word position vectors. The root vector is used to represent the basic form and semantics of the segmentation data, the part-of-speech vector is used to represent the part-of-speech information of the segmentation data, and the word position vector is used to represent the position of each segmentation data in the sentence text. By constructing the segmentation matrix, richer feature information in the sentence text is mentioned, so as to improve the performance and prediction ability of the model when the language processing model is trained later.

[0083] In one embodiment, the process of vectorizing word segmentation data is as follows: Figure 3 The flowchart shown includes the following steps:

[0084] S301, identifying the root information of each word segmentation data, and generating a root vector for each word segmentation data.

[0085] S302, identifying the part-of-speech information of each word segmentation data, and generating a part-of-speech vector for each word segmentation data.

[0086] S303, identifying the word part information of each word segmentation data, and generating a word part vector for each word segmentation data.

[0087] S304, concatenating the root vector, the part-of-speech vector and the word position vector of each word segmentation data to obtain a word segmentation matrix for each word segmentation data.

[0088] For each word segmentation data, the root, part of speech and word position need to be calculated respectively, and then vectorized according to the calculation results. For example, for the root information of each word segmentation data, the root of each word segmentation data can be determined through a pre-built database and the root vector can be calculated; or a deep learning model such as Word2Vec can be used to extract the root information of the word segmentation data and generate a root vector. For the word position information of each word segmentation data, the corresponding word position vector can be generated according to the position information of each word segmentation data in the sentence text. For example, for the sentence text "Who am I", where "I" is in the first position, its corresponding word position vector can be (1, 0, 0); "I am" is in the second position, its corresponding word position vector can be (0, 1, 0); "Who" is in the third position, its corresponding word vector can be (1, 1, 0). For the part-of-speech vector of each word segmentation data, it can be determined according to the part of speech of each word segmentation data in the sentence text. For example, "This is a beautiful flower", where "flower" is a noun. According to the preset correspondence rules between parts of speech and vectors, the part-of-speech vector corresponding to the noun is (1, 1, 1, 0), so this vector is the part-of-speech vector of "flower".

[0089] For the generation process of part-of-speech vectors, please refer to Figure 4 The flowchart shown in the figure has the following specific steps:

[0090] S401, obtaining a preset part-of-speech dictionary.

[0091] S402, identifying the part-of-speech information of each word segmentation data according to the part-of-speech dictionary, and generating the part-of-speech vector.

[0092] In some embodiments, the part-of-speech dictionary includes part-of-speech classification information of multiple word segmentation data. The construction method of the part-of-speech dictionary can be constructed based on the language type processed by the subsequent language processing model. For example, when the language processing model is required to accurately learn and predict Chinese text, the part-of-speech dictionary can include commonly used Chinese words and the part-of-speech classification of each word. After obtaining the part-of-speech dictionary, the word segmentation data and the part-of-speech dictionary are matched to determine the part-of-speech information recorded in the part-of-speech dictionary for each word segmentation data. Finally, according to the correspondence rule between the part-of-speech information and the part-of-speech vector, the part-of-speech vector corresponding to each word segmentation data is determined.

[0093] By adding part-of-speech vectors to the word segmentation matrix of the word segmentation data, the dimension of text representation is enriched, the language processing model's understanding of vocabulary functions is enhanced, and the accuracy of text representation is improved, which in turn helps to improve the learning and prediction capabilities of the language processing model.

[0094] S104, concatenating the word segmentation matrices of the plurality of word segmentation data to obtain a target sentence matrix of the sentence text.

[0095] In some embodiments, after calculating the word segmentation matrix of each word segmentation data in the sentence text, the word segmentation matrices of multiple word segmentation data are concatenated into a final matrix according to the order in the sentence text to obtain the target sentence matrix of the sentence text.

[0096] By comprehensively representing the word segmentation data in the sentence text using root vectors, part-of-speech vectors, and word position vectors, we can more comprehensively and precisely characterize words and their contexts, thereby improving the robustness and accuracy of text representation.

[0097] In one embodiment, the process of concatenating the word segmentation matrix is ​​as follows: Figure 5 The specific steps are as follows:

[0098] S501, determining the sequence information of the plurality of word segmentation data according to the word position vector.

[0099] S502: Arrange the word segmentation matrices of the plurality of word segmentation data according to the sequence information to obtain the target sentence matrix.

[0100] When splicing multiple word segmentation matrices, it is necessary to determine the order information of each word segmentation matrix in the sentence text, so as to splice the multiple word segmentation matrices according to the order information and obtain a complete target sentence matrix for the entire sentence text. According to the word position vector, the order information of multiple word segmentation data in the sentence text can be quickly determined to improve the splicing speed of the word segmentation matrix. After determining the order information of the word segmentation data, the word segmentation matrix of the multiple word segmentation data can be spliced ​​according to the order information. Specifically, each word segmentation matrix can be used as a row matrix, and the multiple word segmentation matrices can be sorted by column according to the order information to obtain the final target sentence vector. In one embodiment, the sentence text is "This is a beautiful flower." The word segmentation matrix in the sentence text is shown in Table 1:

[0101] Table 1

[0102] Word segmentation data Root vector Part of Speech Vector Word vector Word segmentation matrix this (1,0,0,0,0) (1,0,0,0) (1,0,0,0) (1,0,0,0,0)(1,0,0,0)(1,0,0,0) yes (0,1,0,0,0) (1,1,0,0) (0,1,0,0) (0,1,0,0,0)(1,1,0,0)(0,1,0,0) One (0,0,1,0,0) (0,0,1,0) (1,1,0,0) (0,0,1,0,0)(0,0,1,0)(1,1,0,0) Beautiful (0,0,0,1,0) (1,0,1,0) (1,1,1,0) (0,0,0,1,0)(1,0,1,0)(1,1,1,0) Flowers (1,0,0,1,0) (1,1,1,0) (1,1,1,1) (1,0,0,1,0)(1,1,1,0)(1,1,1,1) 。 (1,1,0,1,0) (1,1,1,1) (1,1,0,1) (1,1,0,1,0)(1,1,1,1)(1,1,0,1)

[0103] According to the word segmentation matrices of multiple word segmentation data in Table 1, the final target sentence matrix shown in formula (1) can be directly and accurately concatenated:

[0104]

[0105]

[0106] Among them, the first column represents the root vectors of multiple word segmentation data, the second column represents the part-of-speech vectors of multiple word segmentation data, and the third column represents the word position vectors of multiple word segmentation data.

[0107] It should be noted that the vector length of the part-of-speech vector is related to the classification of the part of speech; the vector length of the word position vector is related to the length of the sentence text. For example, a 512K word position vector can represent a text of 510,000 to 600,000 words.

[0108] Furthermore, when concatenating the word segmentation matrices of multiple word segmentation data, it is necessary to set the coefficients of the root vector, part-of-speech vector, and word position vector in the word segmentation data respectively, so that the language processing model can better learn and predict the sentence text. For specific methods of setting the word segmentation data coefficients, please refer to Figure 6 The flowchart shown includes the following steps:

[0109] S601, arranging the word segmentation matrix according to the sequence information to obtain an initial sentence matrix.

[0110] S602, setting the coefficients of the root column vector, the part-of-speech column vector and the word position column vector in the initial sentence matrix to obtain the target sentence matrix.

[0111] In some embodiments, the initial sentence matrix includes a root column vector, a part-of-speech column vector, and a word position column vector. For example, in formula (1), the first column is a root column vector, the second column is a part-of-speech column vector, and the third column is a word position column vector. After arranging the word segmentation matrices of multiple word segmentation data according to the sequence information of each word segmentation data, the initial sentence matrix can be obtained. At this time, the coefficients of each column vector of the initial sentence matrix are not set. After obtaining the initial sentence matrix, the coefficients of the root column vector, the part-of-speech column vector, and the word position column vector in the initial sentence matrix are set respectively, and the target sentence matrix can be obtained, which can be specifically shown as formula (2):

[0112]

[0113] Among them, α is the coefficient of the root vector, β is the coefficient of the part-of-speech vector, and γ is the coefficient of the word position vector. It should be noted that in order to ensure the training effect of the language processing model, the above three coefficients can be set to 1 at the beginning. The above three coefficients can be tuned by learning and predicting the target sentence matrix through the language processing model.

[0114] In one embodiment, the present disclosure also provides another text representation method flow chart, see Figure 7 , the flowchart includes the following steps:

[0115] S701, obtaining text data for training a language processing model.

[0116] S702, using a word segmentation tool to perform word segmentation calculation on the text data to obtain vocabulary data in the text data.

[0117] S703, calculate the root of each vocabulary data through the deep learning model, and vectorize the root data.

[0118] S704, calculating the part of speech for each vocabulary data using a part of speech tagging tool, and vectorizing the part of speech data.

[0119] S705, calculating the word position of each word data according to the position information of the word in the text data, and vectorizing the word position data.

[0120] S706, concatenating the root vector, part-of-speech vector and word position vector of each word to obtain a vocabulary matrix of each word.

[0121] S707, constructing a text matrix of the text data according to the vocabulary matrix of all the words in the text data, and realizing vectorized representation of the text data.

[0122] Corresponding to the above-mentioned embodiments of the text characterization method, the present disclosure also provides embodiments of a text characterization device.

[0123] Please refer to Figure 8 The present disclosure provides a text representation device block diagram, the device comprising:

[0124] The acquisition unit 801 is used to acquire the input sentence text;

[0125] A word segmentation unit 802 is used to segment the sentence text to obtain a plurality of word segmentation data;

[0126] The processing unit 803 is used to vectorize each word segmentation data to obtain a word segmentation matrix of each word segmentation data, wherein the word segmentation matrix includes a root vector, a part-of-speech vector and a word position vector, and the part-of-speech vector is used to represent the part-of-speech information of the word segmentation data;

[0127] The concatenation unit 804 is used to concatenate the word segmentation matrices of the plurality of word segmentation data to obtain a target sentence matrix of the sentence text.

[0128] In one embodiment, the word segmentation unit 802 is used to:

[0129] Obtaining a predefined word segmentation dictionary, wherein the word segmentation dictionary includes a plurality of known words;

[0130] Matching the sentence text with the word segmentation dictionary to obtain a sentence matching result;

[0131] The sentence text is segmented according to the sentence matching result to obtain the multiple segmentation data.

[0132] In one embodiment, the processing unit 803 is used to:

[0133] Identify the root information of each word segmentation data and generate the root vector of each word segmentation data;

[0134] Identify the part-of-speech information of each word segmentation data and generate a part-of-speech vector for each word segmentation data;

[0135] Identify the word position information of each word segmentation data and generate the word position vector of each word segmentation data;

[0136] The root vector, part-of-speech vector and word position vector of each word segmentation data are concatenated to obtain the word segmentation matrix of each word segmentation data.

[0137] In one embodiment, the processing unit 803 is used to:

[0138] Obtaining a preset part-of-speech dictionary, wherein the part-of-speech dictionary includes part-of-speech classification information of a plurality of word segmentation data;

[0139] According to the part-of-speech dictionary, the part-of-speech information of each word segmentation data is identified and the part-of-speech vector is generated.

[0140] In one embodiment, the splicing unit 804 is used to:

[0141] Determining the sequence information of the plurality of word segmentation data according to the word position vector;

[0142] The word segmentation matrices of the plurality of word segmentation data are arranged according to the sequence information to obtain the target sentence matrix.

[0143] In one embodiment, the splicing unit 804 is used to:

[0144] Arranging the word segmentation matrix according to the sequence information to obtain an initial sentence matrix, wherein the initial sentence matrix includes a root column vector, a part-of-speech column vector, and a word position column vector;

[0145] The coefficients of the root column vector, the part-of-speech column vector and the word position column vector in the initial sentence matrix are set to obtain the target sentence matrix.

[0146] The embodiments of the device represented by the text of the present disclosure can be applied to electronic devices. The device embodiments can be implemented by software, hardware, or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of the electronic device in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory and running them. From the hardware level, such as Fig. 9 As shown, it is a hardware structure diagram of the electronic device where the text characterization device of the present disclosure is located, except Fig. 9 In addition to the processor, memory, network interface, and non-volatile memory shown, the electronic device in which the device in the embodiment is located may also include other hardware according to the actual function of the electronic device, which will not be described in detail.

[0147] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0148] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can refer to the partial description of the method embodiments. The device embodiments described above are only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the disclosed solution. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0149] Based on the same inventive concept, the present disclosure also provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of any of the text representation methods are implemented.

[0150] Based on the same inventive concept, the present disclosure also provides a computer program product, which implements the steps of any of the text representation methods when executed by a processor.

[0151] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the particular order or sequential order shown to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

[0152] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure shall be included in the scope of protection of the present disclosure.

Claims

1. A text representation method, characterized in that: The method comprises: Get the input statement text; Segmenting the sentence text to obtain a plurality of segmentation data; Vectorize each word segmentation data to obtain a word segmentation matrix for each word segmentation data, wherein the word segmentation matrix includes a root vector, a part-of-speech vector and a word position vector, and the part-of-speech vector is used to represent the part-of-speech information of the word segmentation data; The word segmentation matrices of the plurality of word segmentation data are concatenated to obtain a target sentence matrix of the sentence text.

2. The method according to claim 1, characterized in that The sentence text is segmented to obtain a plurality of segmentation data, including: Obtaining a predefined word segmentation dictionary, wherein the word segmentation dictionary includes a plurality of known words; Matching the sentence text with the word segmentation dictionary to obtain a sentence matching result; The sentence text is segmented according to the sentence matching result to obtain the multiple segmentation data.

3. The method according to claim 1, characterized in that The vectorization of each word segmentation data to obtain a word segmentation matrix for each word segmentation data includes: Identify the root information of each word segmentation data and generate the root vector of each word segmentation data; Identify the part-of-speech information of each word segmentation data and generate a part-of-speech vector for each word segmentation data; Identify the word position information of each word segmentation data and generate the word position vector of each word segmentation data; The root vector, part-of-speech vector and word position vector of each word segmentation data are concatenated to obtain the word segmentation matrix of each word segmentation data.

4. The method according to claim 3, characterized in that The identifying the part-of-speech information of each word segmentation data and generating the part-of-speech vector of each word segmentation data includes: Obtaining a preset part-of-speech dictionary, wherein the part-of-speech dictionary includes part-of-speech classification information of a plurality of word segmentation data; According to the part-of-speech dictionary, the part-of-speech information of each word segmentation data is identified and the part-of-speech vector is generated.

5. The method according to claim 1, characterized in that The step of concatenating the segmentation matrices of the plurality of segmentation data to obtain a target sentence matrix of the sentence text includes: Determining the sequence information of the plurality of word segmentation data according to the word position vector; The word segmentation matrices of the plurality of word segmentation data are arranged according to the sequence information to obtain the target sentence matrix.

6. The method according to claim 5, characterized in that The step of arranging the word segmentation matrices of the plurality of word segmentation data according to the sequence information to obtain a target sentence matrix of the sentence text includes: Arranging the word segmentation matrix according to the sequence information to obtain an initial sentence matrix, wherein the initial sentence matrix includes a root column vector, a part-of-speech column vector, and a word position column vector; The coefficients of the root column vector, the part-of-speech column vector and the word position column vector in the initial sentence matrix are set to obtain the target sentence matrix.

7. A text representation device, characterized in that: The device comprises: An acquisition unit, used for acquiring an input sentence text; A word segmentation unit, used for segmenting the sentence text to obtain a plurality of word segmentation data; A processing unit, used to vectorize each word segmentation data to obtain a word segmentation matrix of each word segmentation data, wherein the word segmentation matrix includes a root vector, a part-of-speech vector and a word position vector, and the part-of-speech vector is used to represent the part-of-speech information of the word segmentation data; The concatenation unit is used to concatenate the word segmentation matrices of the plurality of word segmentation data to obtain a target sentence matrix of the sentence text.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that When the program product is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images