Text compression method and device, equipment and storage medium

Abstract

The invention provides a text compression method and device, equipment and a storage medium, and the method comprises the steps: carrying out the word segmentation of an obtained original text, obtaining at least one text word segmentation result, obtaining a combination matrix of each text word segmentation result, and enabling the combination matrix to be determined according to at least one of a word segmentation matrix, a word segmentation feature matrix and a word segmentation position matrix. And performing compression processing on each combined matrix by using a plurality of preset compression matrixes to obtain target matrixes, merging the target matrixes corresponding to each combined matrix to obtain a merged matrix, determining target candidate words of each row of vectors of the merged matrix from a preset dictionary, and merging all the target candidate words to obtain a compressed text. By performing multi-dimensional representation on the original text, the core knowledge point understanding accuracy of the text can be improved, and then the text compression accuracy and compression rate and the meaning integrity of the original text content can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, and particularly relates to a text compression method and device, equipment and a storage medium. BACKGROUND

[0002] In recent years, deep learning technology has been widely applied to natural language processing (NLP), and especially based on neural network models such as recurrent neural network (RNN), long short-term memory (LSTM), etc., which effectively improves the understanding ability of the language model.

[0003] With the continuous expansion of the model size, in order to cope with the challenges of computational complexity and storage requirements, it is usually necessary to compress the context information in the model to reduce the amount of data processed by the model.

[0004] However, the current text compression method often only considers the single-dimensional information of the text, and this single-dimensional processing method will lead to the loss and misunderstanding of key information, affecting the accurate understanding of the core knowledge points of the text, and further leading to low accuracy, compression rate and meaning integrity of the text content. SUMMARY

[0005] Therefore, the present application provides a text compression method, device, equipment and storage medium to solve the problems in the related art.

[0006] In a first aspect of the present application, a text compression method is provided, which comprises:

[0007] segmenting the obtained original text to obtain at least one text segmentation result;

[0008] obtaining a combination matrix of each text segmentation result, wherein the combination matrix is determined according to at least one of the segmentation matrix, the segmentation feature matrix and the segmentation position matrix;

[0009] performing compression processing on each combination matrix using a plurality of preset compression matrices to obtain a target matrix, wherein each compression matrix corresponds to a word attribute;

[0010] merging the target matrix corresponding to each combination matrix to obtain a merged matrix;

[0011] for each row vector in the merged matrix, determining the probability of each word in the preset dictionary as a candidate word through the row vector, and taking the candidate word with a probability meeting the requirements as the target candidate word of the row vector;

[0012] merge all target candidate words to obtain the compressed text.

[0013] According to an embodiment of the present application, the method further comprises:

[0014] For each segmentation in the text segmentation result, according to semantic information of the segmentation, determining a root vector of the segmentation;

[0015] According to part-of-speech information of the segmentation, determining a part-of-speech vector of the segmentation;

[0016] According to position information of the segmentation in the text segmentation result, determining a word position vector of the segmentation;

[0017] Combining the root vector, the part-of-speech vector and the word position vector of all segmentations in the text segmentation result respectively to obtain a segmentation matrix, a segmentation feature matrix and a segmentation position matrix of the text segmentation result.

[0018] According to an embodiment of the present application, the determining the combination matrix according to the segmentation matrix, the segmentation feature matrix and the segmentation position matrix comprises:

[0019] Linearly transforming the segmentation matrix, the segmentation feature matrix and the segmentation position matrix respectively by using corresponding first weight matrices and first bias terms, the first weight matrices and the first bias terms being obtained by pre-training;

[0020] Weighted summing results of the linear transformation to obtain the combination matrix.

[0021] According to an embodiment of the present application, the compressing each combination matrix by using a plurality of preset compression matrices to obtain a target matrix comprises:

[0022] For each combination matrix, compressing the combination matrix by using each compression matrix and a displacement matrix corresponding to the compression matrix;

[0023] Merging results of the compression to obtain a target matrix corresponding to the combination matrix.

[0024] According to an embodiment of the present application, after compressing each combination matrix by using a plurality of preset compression matrices to obtain a target matrix, the method further comprises:

[0025] Determining a similarity between the combination matrix and the target matrix;

[0026] Determining a first correlation between the original text vector and the combination matrix, and determining a second correlation between the original text vector and the target matrix;

[0027] determining a difference between the first similarity and the second similarity;

[0028] in a case where the similarity is greater than a first preset threshold and the difference is less than a second preset threshold, compressing the target matrix again using the plurality of preset compression matrices to update the target matrix.

[0029] According to an embodiment of the present application, the merging of the target matrix corresponding to each combination matrix to obtain a merged matrix comprises:

[0030] linearly transforming the target matrix corresponding to each combination matrix using a corresponding second weight matrix and a second bias term, respectively, the second weight matrix and the second bias term being obtained through pre-training;

[0031] merging the results of the linear transformation to obtain the merged matrix.

[0032] According to an embodiment of the present application, the determining of the probability of each word in a preset dictionary as a candidate word through the row vector comprises:

[0033] obtaining a vector distance between the row vector and a word vector corresponding to each word in the preset dictionary;

[0034] normalizing all the obtained vector distances to obtain the probability of each word in the preset dictionary as a candidate word.

[0035] In a second aspect of the present application, a text compression device is provided, which comprises:

[0036] a word segmentation unit configured to segment the obtained original text to obtain at least one text segmentation result;

[0037] an obtaining unit configured to obtain a combination matrix of each text segmentation result, the combination matrix being determined according to at least one of a word segmentation matrix, a word segmentation feature matrix, and a word segmentation position matrix;

[0038] a compression unit configured to compress each combination matrix using a plurality of preset compression matrices to obtain a target matrix, wherein each compression matrix corresponds to a vocabulary attribute;

[0039] a merging unit configured to merge the target matrix corresponding to each combination matrix to obtain a merged matrix;

[0040] a determining unit configured to, for each row vector in the merged matrix, determine a probability of each word in a preset dictionary as a candidate word through the row vector, and take a candidate word whose probability meets a requirement as a target candidate word of the row vector;

[0041] The output unit is configured to merge all the target candidate words to obtain the compressed text.

[0042] In a third aspect of the present application, an electronic device is provided, comprising a processor and a memory, the memory storing machine executable instructions capable of being executed by the processor, and the processor is configured to execute the machine executable instructions to implement the steps of the method proposed in the above embodiments.

[0043] In a fourth aspect of the present application, a machine readable storage medium is provided, the machine readable storage medium storing machine executable instructions, and the machine executable instructions are executed by a processor to implement the steps of the method proposed in the above embodiments.

[0044] In a fifth aspect of the present application, a computer program product is provided, comprising computer programs / instructions, and the computer programs / instructions are executed by a processor to implement the steps of the method proposed in the above embodiments.

[0045] As can be seen from the above technical solutions, by performing word segmentation on the obtained original text to obtain at least one text segmentation result, obtaining a combination matrix of each text segmentation result, the combination matrix being determined according to at least one of a segmentation matrix, a segmentation feature matrix and a segmentation position matrix, performing compression processing on each combination matrix by using a plurality of preset compression matrices to obtain a target matrix, merging the target matrix corresponding to each combination matrix to obtain a merged matrix, determining a target candidate word of each row vector of the merged matrix from a preset dictionary, and merging all the target candidate words to obtain a compressed text. By performing multi-dimensional representation on the original text, the accuracy of understanding the core knowledge points of the text can be improved, and thus the accuracy, compression rate and meaning integrity of the original text content of the text compression can be improved.

[0046] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 is a flowchart of a text compression method provided by an embodiment of the present application;

[0048] Figure 2 is a flowchart of obtaining a combination matrix provided by an embodiment of the present application;

[0049] Figure 3 is a flowchart of compression processing provided by an embodiment of the present application;

[0050] Figure 4 is a flowchart of a text compression method provided by an embodiment of the present application;

[0051] Figure 5 is a structural schematic diagram of a text compression device provided by an embodiment of the present application.

[0052] Figure 6 is a hardware structure schematic diagram of an electronic device shown by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0053] The exemplary embodiments will be described in detail herein with reference to the attached drawings. The following description is with reference to the drawings, in which like numerals refer to like elements throughout. The embodiments described in the following exemplary embodiments are not meant to represent all embodiments in accordance with the present application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present application as detailed in the appended claims.

[0054] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used in this application and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0055] In order to make the technical solutions provided by the embodiments of the present application better understood by those skilled in the art, and to make the above-mentioned purposes, features and advantages of the embodiments of the present application more apparent and easy to understand, the technical solutions in the embodiments of the present application are further described in detail below with reference to the drawings.

[0056] In natural language processing (NLP), accurately understanding the user's intention is the key to achieving efficient human-computer interaction. In recent years, deep learning technology has been widely applied in natural language processing, especially neural network-based models such as recurrent neural network (RNN), long short-term memory (LSTM), etc., which effectively improve the understanding ability of language models.

[0057] As the model size continues to expand, the amount of context information it needs to process also increases, which directly leads to an increase in computational complexity and an explosion in storage requirements. In the face of this challenge, in order to reduce the computational burden and storage pressure while ensuring model performance, it is usually necessary to compress the context in the model, that is, to extract and retain key information and core concepts in the context, and to eliminate redundant data, in order to reduce the amount of data that the model needs to process.

[0058] However, current text compression methods often only consider single-dimensional information of the text, such as vocabulary, grammatical structure, etc., without considering the interaction and complementarity between different-dimensional information (such as semantic, emotional, and context information, etc.). This single-dimensional processing method may cause key information in the text to be omitted or misunderstood during the text compression process, affecting the accuracy of understanding the core knowledge points of the text, and further resulting in low accuracy, compression rate, and integrity of the meaning of the text content.

[0059] In view of this, embodiments of the present application disclose a text compression method to solve the deficiencies in the related art.

[0060] As Figure 1 shown, Figure 1 is a flowchart of a text compression method provided by an embodiment of the present application. The text compression method may include the following steps:

[0061] S101: Segment the obtained original text to obtain at least one text segmentation result.

[0062] Text segmentation is the process of recombining the original text, that is, a continuous sequence of characters, into a sequence of words according to certain rules, algorithms, or models. The original text here may include a sequence of Chinese characters, a sequence of English characters, a sequence of characters in other languages, or a mixed language sequence, etc.

[0063] For example, assume the original text is "The weather is really nice today". The original text can be input into a pre-trained text segmentation model. This text segmentation model can segment the original text based on various factors, such as the scenarios where the vocabulary appears (such as education, economy, society, etc.) and the usage frequency of the vocabulary (including the frequency of word formation and the statistical probability of the vocabulary appearance).

[0064] Since the language usage habits and vocabulary collocations in different scenarios may be different, the text segmentation model can perform multi-level and multi-angle text segmentation operations on the same original text based on various factors, so as to output one or more text segmentation results. For example, in one iteration, the text segmentation result may be "Today | weather | really | nice", and in another iteration, the text segmentation result may be "Today | weather | really nice", in this text segmentation result, "really nice" is a segmentation and is regarded as a whole phrase.

[0065] It should be noted that the original text can be an extremely long text containing multiple sentences. Here, taking the original text as a single sentence for example is for the convenience of understanding and explanation.

[0066] In some embodiments, a query text in a dialogue system and a context of the query text in the dialogue system can be obtained, and according to the query text and the context of the query text, an original text of a preset size can be obtained, and the obtained original text can be segmented to obtain at least one text segmentation result.

[0067] Specifically, the obtained query text can be vectorized to obtain a query text vector.

[0068] The context of the obtained query text can be converted into a plurality of context vectors. The context vector is a vector representation of each element in the context of the query text, and the element can include a word, a phrase or other meaningful content in the context.

[0069] The similarity between each context vector and the query text vector can be calculated. It should be noted that the embodiments of the present application do not specifically limit the method of calculating the similarity, for example, the similarity can be calculated by cosine similarity or Euclidean distance.

[0070] The contexts corresponding to the top K context vectors with the highest similarity can be combined to obtain the original text, wherein the sum of the lengths of the contexts corresponding to the K context vectors is the preset size. For example, the preset size can be 128K, 512K, etc.

[0071] In some embodiments, the obtained original text can be segmented to obtain a set number of text segmentation results. For example, the set number can be 2, 3, etc.

[0072] In some embodiments, the obtained original text can be segmented using a preset segmentation rule to obtain at least one text segmentation result.

[0073] Specifically, a large amount of historical text data can be obtained, and the historical text data includes text data of different fields and scenarios.

[0074] According to the historical text data, a preset segmentation rule can be generated. Specifically, the historical text data can be analyzed and counted, including word frequency statistics, scene statistics and boundary vocabulary statistics, etc. The word frequency statistics can include word group frequency, appearance frequency of vocabulary, etc., the scene statistics can include education, economy, society, etc., and finally the preset segmentation rule can be generated according to the statistical analysis result.

[0075] The obtained original text can be segmented using the preset segmentation rule to obtain at least one text segmentation result.

[0076] It should be noted that the embodiments of the present application do not specifically limit the method of "segmenting the obtained original text", for example, deep learning, machine learning or hybrid method can also be used for segmentation.

[0077] In the embodiments of the present application, by performing word segmentation on the obtained original text, at least one text segmentation result is obtained, each text segmentation result can be a potential understanding of the original text, so as to improve the accuracy of understanding the core knowledge points of the original text, and then improve the accuracy, compression rate and meaning integrity of the original text content in the subsequent text compression step.

[0078] S102: Obtain a combination matrix of each text segmentation result, wherein the combination matrix is determined according to at least one of the segmentation matrix, the segmentation feature matrix and the segmentation position matrix.

[0079] For each text segmentation result, a combination matrix of the text segmentation result is obtained, wherein the combination matrix is determined according to at least one of the segmentation matrix, the segmentation feature matrix and the segmentation position matrix.

[0080] In some embodiments, for each text segmentation result, each segmentation in the text segmentation result can be vectorized to obtain a vector of each segmentation, and the vectors of all segmentations in the text segmentation result are combined to obtain a segmentation matrix of the text segmentation result. The purpose of vectorization is to enable the computer to recognize and perform mathematical calculations.

[0081] For example, for the text segmentation result "today | weather | really | not good", the vector of the segmentation "today" can be obtained as (0, 0, 1), the vector of the segmentation "weather" can be obtained as (0, 0, 2), the vector of the segmentation "really" can be obtained as (0, 0, 3), and the vector of the segmentation "not good" can be obtained as (0, 0, 4). All segmentation vectors are combined to obtain a segmentation matrix, which is shown as follows:

[0082]

[0083] In some embodiments, each segmentation in the text segmentation result can be vectorized using a preset vocabulary table, wherein each entry in the preset vocabulary table includes a segmentation and a unique word vector pre-assigned to the segmentation.

[0084] Specifically, for each segmentation in the text segmentation result, the corresponding entry of the segmentation is searched in the preset vocabulary table, and the word vector in the entry is used as the vector of the segmentation.

[0085] It should be noted that the method of vectorization in the embodiments of the present application is not specifically limited, for example, vectorization can be performed by word embedding, one-hot encoding, etc.

[0086] In some embodiments, for each text segmentation result, target features of each segmentation in the text segmentation result are extracted, and the target features are vectorized to obtain a feature vector of each segmentation. The feature vectors of all segmentations are combined to obtain a segmentation feature matrix of the text segmentation result.

[0087] For example, the target features can include semantic categories, parts of speech, and emotions, etc.

[0088] In some embodiments, a label profile of each segmentation in the text segmentation result can be obtained from a preset vocabulary table, and each segmentation in the text segmentation result is vectorized according to a predefined mapping relationship between labels and vectors.

[0089] Specifically, the preset vocabulary table contains label profiles corresponding to multiple segmentations. For example, the label profile can be: (

today

weather

[0090] The dimension of the feature vector of the segmentation is predefined, each dimension corresponds to a label category, and a mapping relationship between labels and vectors is constructed.

[0091] For example, assuming that the preset vocabulary table contains the following several label categories: semantic categories, parts of speech, and emotions, then the dimension of the feature vector of the segmentation can be defined as 3 dimensions.

[0092] The mapping between labels and vectors is constructed, which can be as follows:

[0093] {“time”: (1, 0, 0); “weather”: (2, 0, 0); “tone”: (3, 0, 0); “evaluation”: (4, 0, 0); “noun”: (0, 1, 0); “adjective”: (0, 2, 0); “adverb”: (0, 3, 0); “happy”: (0, 0, 1); “ordinary”: (0, 0, 2); …}.

[0094] For the text segmentation result “today | weather | really | not bad”, the label profile “

today

weather

[0095] The feature vectors of all segmentations are combined to obtain a segmentation feature matrix of the text segmentation result. For example, the segmentation feature matrix can be as follows:

[0096]

[0097] In some embodiments, for each text segmentation result, the position of each segmentation in the text segmentation result is vectorized to obtain a position vector of each segmentation, and the position vectors of all segmentations in the text segmentation result are combined to obtain a segmentation position matrix of the text segmentation result.

[0098] For example, the position of a segmentation can be the absolute position of the segmentation in a text segmentation result.

[0099] In some embodiments, for each segmentation in each text segmentation result, a root vector of the segmentation is determined according to semantic information of the segmentation.

[0100] According to the part-of-speech information of the segmentation, a part-of-speech vector of the segmentation is determined.

[0101] According to the position information of the segmentation in the text segmentation result, a word position vector of the segmentation is determined.

[0102] The root vectors, part-of-speech vectors and word position vectors of all segmentations in the text segmentation result are combined respectively to obtain a segmentation matrix, a segmentation feature matrix and a segmentation position matrix of the text segmentation result.

[0103] Specifically, for each segmentation in each text segmentation result, semantic information of the segmentation can be obtained. It should be noted that the embodiments of the present application do not specifically limit the method of "obtaining semantic information of the segmentation". For example, a dictionary-based method, a preset dictionary contains the semantic information of the segmentation corresponding to the segmentation, and the semantic information of the segmentation can be obtained by matching the segmentation in the preset dictionary, or a deep learning-based method, the segmentation is input into a model pre-trained using a large corpus, and the semantic information of the segmentation is output.

[0104] According to the semantic information of the segmentation, a root vector of the segmentation is determined. It should be noted that the embodiments of the present application do not specifically limit the method of "determining the root vector of the segmentation according to the semantic information of the segmentation", for example, a machine learning-based method, the semantic information of the segmentation is input into a pre-trained word embedding model (such as Word2vec, GloVe, etc.), and the root vector of the segmentation is output, or a self-defined mapping function, which maps the semantic information of the segmentation to a vector in a high-dimensional space.

[0105] The root vectors of all segmentations in the text segmentation result are combined to obtain a segmentation matrix of the text segmentation result, wherein each row of the segmentation matrix represents a root vector of a segmentation.

[0106] For each word segmentation in each text segmentation result, the part-of-speech information of the word segmentation can be obtained. It should be noted that the embodiments of the present application do not specifically limit the method of "obtaining the part-of-speech information of the word segmentation", for example, the part-of-speech information of the word segmentation can be obtained based on a part-of-speech tagging tool, a pre-defined rule, deep learning or machine learning, etc.

[0107] According to the part-of-speech information of the word segmentation, the part-of-speech vector of the word segmentation is determined. It should be noted that the embodiments of the present application do not specifically limit the method of "determining the part-of-speech vector of the word segmentation according to the part-of-speech information of the word segmentation", for example, the part-of-speech vector of the word segmentation can be determined based on a pre-trained word embedding model, a self-defined mapping function or a statistical method, etc.

[0108] The part-of-speech vectors of all word segmentations in the text segmentation result are combined to obtain the word segmentation feature matrix of the text segmentation result, wherein each row of the word segmentation feature matrix represents the part-of-speech vector of a word segmentation.

[0109] For each word segmentation in each text segmentation result, the absolute position of the word segmentation in the text segmentation result can be obtained, and the absolute position of the word segmentation is converted into a numerical vector to obtain the word position vector of the word segmentation. The word position vectors of all word segmentations in the text segmentation result are combined to obtain the word position matrix of the text segmentation result, wherein each row of the word position matrix represents the word position vector of a word segmentation.

[0110] In some embodiments, the combined matrix can be determined according to the word segmentation matrix, the word segmentation feature matrix and the word segmentation position matrix.

[0111] Specifically, the word segmentation matrix, the word segmentation feature matrix and the word segmentation position matrix are respectively linearly transformed by using a corresponding first weight matrix and a first bias term, and the first weight matrix and the first bias term are obtained by pre-training.

[0112] The results of the linear transformation are summed to obtain the combined matrix.

[0113] The weight matrix W is part of the model parameters, which is a parameter matrix used to convert the input features, and it can map the input features (such as word vectors, position encoding, etc.) from the original feature space to a new feature space.

[0114] The bias term B is also part of the model parameters, which is used to adjust the output after the weight matrix conversion, and can be regarded as a non-linear factor introduced after the linear transformation, which can increase the robustness of the network model.

[0115] As shown in FIG. 1, Figure 2 is a flowchart of obtaining a combined matrix provided by an embodiment of the present application. Figure 2

[0116] ​Specifically, the first weight matrix 1 W x includes 1 W1, 1 W2 and 1 W3, which are weight matrices corresponding to the word segmentation matrix Q1, the word segmentation feature matrix Q2 and the word segmentation position matrix Q3 respectively, and the first bias term 1 B x includes 1 B1, 1 B2 and 1 B3, which are bias terms corresponding to the word segmentation matrix Q1, the word segmentation feature matrix Q2 and the word segmentation position matrix Q3 respectively.

[0117] The linear transformation is performed on the word segmentation matrix Q1, the word segmentation feature vector Q2 and the word segmentation position vector Q3 respectively by using formula (1) to obtain the linear transformation results Q'1, Q'2 and Q'3. Formula (1) is as follows:

[0118] Q x ′= 1 W x ·Q x + 1 B x (1)

[0119] wherein x = 1, 2, 3.

[0120] The linear transformation results are summed to obtain the combination vector ExQ.

[0121] The first weight matrix 1 W x and the first bias term 1 B x are obtained through pre-training. Specifically, before the pre-training starts, the first weight matrix 1 W x and the first bias term 1 B x are randomly initialized. During the pre-training process, the input data is forward propagated through the model to obtain the predicted output, and the difference between the predicted output and the real output is calculated through the loss function to obtain the loss value. According to the obtained loss value, the gradient of the loss with respect to the first weight matrix 1 Wxand the first bias term 1 Bx is calculated using the backpropagation algorithm. Using the calculated gradient, the first weight matrix 1 Wxand the first bias term 1Bx is updated, the above steps are repeated, and the first weight matrix is optimized through multiple iterations 1 Wx and the first bias term 1 Bx until the loss of the dialogue system on the training data is reduced to an acceptable range or a predetermined number of iterations is reached.

[0122] In the embodiments of the present application, by obtaining a combination matrix of each text segmentation result, the combination matrix is determined according to at least one of the segmentation matrix, the segmentation feature matrix, and the segmentation position matrix, by constructing a multi-dimensional representation of the original text, the original text is decomposed into multiple dimensions that are related to each other but independent, thereby improving the accuracy of understanding the core knowledge points of the text, and further improving the accuracy, compression rate and meaning integrity of the original text content in the subsequent text compression step.

[0123] S103: compress each combination matrix using a plurality of preset compression matrices to obtain a target matrix, wherein each compression matrix corresponds to a word attribute.

[0124] A plurality of compression matrices are pre-set, and each compression matrix corresponds to a word attribute. That is, each compression matrix is designed for different word attributes and is specifically used to compress words with specific word attributes.

[0125] For example, the word attribute can include part of speech (such as noun, verb, adjective, etc.), semantic category (such as person, place, time, etc.), or emotional color (such as positive, negative, neutral, etc.), and the like. The embodiments of the present application are not specifically limited in this regard.

[0126] In some embodiments, compressing each combination matrix using a plurality of preset compression matrices to obtain a target matrix includes:

[0127] For each combination matrix, each compression matrix and the displacement matrix corresponding to the compression matrix are used to compress the combination matrix;

[0128] The results of the compression processing are merged to obtain the target matrix corresponding to the combination matrix.

[0129] As shown in Figure 3 , the flowchart of the compression processing provided by the embodiments of the present application is shown in Figure 3 .

[0130] Specifically, for the preset N compression matrices 2 W x , there is a corresponding displacement matrix 2 B x , x = 1, 2,..., N. The displacement matrix 2 B xRobustness of the network model can be increased.

[0131] Exemplarily, N can be 32, 64, or 128, etc.

[0132] Compressed matrix 2 W x and displacement matrix 2 B x is obtained through pre-training, and specific obtaining steps are the same as those of the first weight matrix 1 W x and the first bias term 1 B x , which will not be described herein again.

[0133] Each compressed matrix 2 W x and the displacement matrix 2 B x corresponding to the compressed matrix are used to compress the combination matrix ExQ, to obtain a compressed result ExQ' x , x = 1, 2, …, N. The specific formula is as follows:

[0134] ExQ' x = 2 W x ·ExQ+ 2 B x (2)

[0135] The compressed result ExQ' x is logically added to obtain a target matrix corresponding to the combination matrix ExQ.

[0136] In some embodiments, after the combination matrix is compressed by using the plurality of preset compressed matrices to obtain the target matrix, the method further includes:

[0137] Determining a similarity between the combination matrix and the target matrix.

[0138] The similarity is an index for measuring the closeness of two or more objects in some characteristics or attributes. When two objects exhibit similar characteristics in multiple dimensions, it can be considered that they have a higher similarity. It should be noted that the method for calculating the similarity is not specifically limited in the embodiments of the present application. For example, the similarity can be calculated based on cosine similarity, Euclidean distance, etc.

[0139] Determining a first correlation between the original text vector and the combination matrix, and determining a second correlation between the original text vector and the target matrix.

[0140] Correlation is a statistical measure of the strength and direction of the linear relationship between two variables, emphasizing whether there is a statistical association between the two variables. It should be noted that the embodiments of the present application do not specifically limit the method of calculating the correlation, for example, the similarity can be calculated based on Pearson correlation coefficient, Spearman rank correlation coefficient and the like.

[0141] A correlation difference between the first correlation and the second correlation is determined.

[0142] In a case where the similarity is greater than a first preset threshold and the correlation difference is less than a second preset threshold, the target matrix is compressed again using the plurality of preset compression matrices to update the target matrix.

[0143] Specifically, if the similarity is greater than the first preset threshold and the correlation difference is less than the second preset threshold, the target matrix is compressed again using the plurality of preset compression matrices to obtain a new target matrix, until the similarity of the obtained target matrix is less than or equal to the first preset threshold or the correlation difference is greater than or equal to the second preset threshold, and the target matrix is no longer compressed.

[0144] S104: merging the target matrix corresponding to each combined matrix to obtain a merged matrix.

[0145] After obtaining the target matrix corresponding to each combined matrix, the target matrix is merged to obtain a merged matrix.

[0146] In some embodiments, the target matrix corresponding to each combined matrix is linearly transformed using a corresponding second weight matrix and a second bias term, respectively, the second weight matrix and the second bias term being obtained by pre-training;

[0147] The results of the linear transformation are merged to obtain the merged matrix.

[0148] Specifically, each segmentation method can correspond to a second weight matrix 3 W i and a second bias term 3 B i The second weight matrix 3 W i and the second bias term 3 B i are obtained by pre-training, and the specific obtaining steps are the same as those of the first weight matrix 1 W x and the first bias term 1 B x , which will not be described here.

[0149] Different segmentation methods can obtain different text segmentation results, each text segmentation result corresponds to a combination matrix, and each combination matrix corresponds to a target matrix. For each target matrix Y i , a linear transformation is performed using a corresponding second weight matrix 3 W i and a second bias term 3 B i to obtain a linearly transformed result Y' i . The specific formula is as follows:

[0150] Y' i = 3 W i ·Y i + 3 B i (3)

[0151] The linearly transformed result Y' i is merged to obtain the merged matrix FinalY.

[0152] For example, the original text "today the weather is really good" is segmented to obtain three text segmentation results, and then FinalY=Y'1+Y'2+Y'3.

[0153] S105: For each row vector in the merged matrix, determine the probability of each word in the preset dictionary as a candidate word through the row vector, and take the candidate word with a probability meeting the requirements as the target candidate word of the row vector.

[0154] For example, the probability meeting the requirements can be the highest probability.

[0155] In some embodiments, the vector distance between the row vector and the word vector corresponding to each word in the preset dictionary can be obtained.

[0156] All obtained vector distances are normalized to obtain the probability of each word in the preset dictionary as a candidate word.

[0157] Specifically, the preset dictionary is a collection of a large number of vocabularies, which are predefined and cover all or most commonly used vocabularies in multiple fields or general language. Each word is an entry in the preset dictionary, and each word has a corresponding word vector.

[0158] The distance between the row vector and the word vector corresponding to each word in the preset dictionary is calculated, which can be Euclidean distance or Manhattan distance, etc.

[0159] All calculated vector distances are normalized so that they can be converted into probability values, thereby obtaining the probability of each word in the preset dictionary being a candidate word. The embodiments of this application do not specifically limit the normalization method; for example, normalization can be performed using methods such as the normalization exponential function Softmax or Logistic Regression.

[0160] It should be noted that the embodiments of this application do not specifically limit the method of "determining the probability of each word in the preset dictionary as a candidate word through row vectors". For example, it can also be based on radial basis function, negative sampling or neural network methods.

[0161] In some embodiments, it can be first determined whether each row vector in the merge matrix is ​​a zero vector. If the row vector is a zero vector, the target candidate word of the row vector is determined to be an empty character. If the row vector is not a zero vector, the probability of each word in the preset dictionary as a candidate word is determined through the row vector, and the candidate words whose probabilities meet the requirements are taken as the target candidate words of the row vector.

[0162] S106: Merge all target candidate words to obtain the compressed text.

[0163] The target candidate words corresponding to all row vectors are merged to obtain the compressed text.

[0164] In the embodiments of this application, the acquired original text is segmented to obtain at least one text segmentation result. A combination matrix of each text segmentation result is obtained, which is determined based on at least one of a segmentation matrix, a segmentation feature matrix, and a segmentation position matrix. Each combination matrix is ​​compressed using multiple preset compression matrices to obtain a target matrix. The target matrices corresponding to each combination matrix are merged to obtain a merge matrix. Target candidate words for each row vector of the merge matrix are determined from a preset dictionary. All target candidate words are merged to obtain the compressed text. By performing multi-dimensional representation of the original text, the accuracy of understanding the core knowledge points of the text can be improved, thereby improving the accuracy of text compression, the compression ratio, and the semantic completeness of the original text content.

[0165] like Figure 4 As shown, Figure 4 This is a flowchart illustrating a text compression method provided in an embodiment of this application.

[0166] The acquired original text is segmented into words to obtain at least one word segmentation result. For example, if the acquired original text is "Today the weather is really nice", segmenting the original text into words can yield word segmentation results such as "Today|weather|really|nice" and "Today|weather|really|nice", etc.

[0167] It should be noted that the original text can be an extremely long text containing multiple sentences. Here, the original text is used as a single sentence as an example for ease of understanding and explanation.

[0168] For each text segmentation result, each segmented word in the text segmentation result is vectorized to obtain the vector of each segmented word. The vectors of all segmented words in the text segmentation result are combined to obtain the segmentation matrix Q1 of the text segmentation result.

[0169] The target features are extracted from each word in the text segmentation result, and the target features are vectorized to obtain the feature vector of each word. The feature vectors of all words are combined to obtain the word segmentation feature matrix Q2 of the text segmentation result.

[0170] The position vector of each word in the text segmentation result is vectorized to obtain the position vector of each word. The position vectors of all words in the text segmentation result are combined to obtain the word segmentation position matrix Q3 of the text segmentation result.

[0171] For the word segmentation matrix Q1, word segmentation feature matrix Q2, and word segmentation position matrix Q3, respectively, use the corresponding first weight matrix 1 W x and the first bias term 1 B x Perform a linear transformation on x = 1, 2, 3. The result of the linear transformation is Q'1 = 1 W1×Q1+ 1 B1, Q'2 = 1 W2×Q2+ 1 B2, Q'3 = 1 W3×Q3+ 1 B3.

[0172] Summing the results Q'1, Q'2, and Q'3 from the linear transformation yields the combination matrix ExQ. Where ExQ = Q'1 + Q'2 + Q'3.

[0173] N compression matrices are pre-trained, each corresponding to a specific lexical attribute. That is, each compression matrix is ​​designed for different lexical attributes and is specifically used to compress words with specific lexical attributes.

[0174] Using each compression matrix 2 W x The displacement matrix corresponding to this compression matrix 2 B x The combined matrix ExQ is compressed to obtain the compressed result ExQ'. x x = 1, 2, ..., N.

[0175] The compressed result ExQ'x performing logical addition to obtain a target matrix corresponding to the combination matrix ExQ.

[0176] determining the similarity between the combination matrix ExQ and the target matrix.

[0177] determining a first correlation between the original text vector and the combination matrix ExQ, determining a second correlation between the original text vector and the target matrix, and determining a correlation difference between the first correlation and the second correlation.

[0178] If the similarity is greater than a first preset threshold and the correlation difference is less than a second preset threshold, the target matrix is compressed again using a plurality of preset compression matrices to obtain a new target matrix, until the similarity corresponding to the obtained target matrix is less than or equal to the first preset threshold or the correlation difference is greater than or equal to the second preset threshold, and the target matrix is no longer compressed.

[0179] respectively, using a corresponding second weight matrix 3 W i and a second bias term 3 B i performing linear transformation to obtain a linear transformation result Y' i .

[0180] combining all linear transformation results Y' i to obtain the merging matrix FinalY.

[0181] For each row vector in the merging matrix FinalY, the distance between the row vector and the word vector corresponding to each word in the preset dictionary is calculated, all vector distances obtained by calculation are normalized so that these vector distances can be converted into probability values, thereby obtaining the probability of each word in the preset dictionary as a candidate word, and the candidate word with the highest probability is taken as the target candidate word of the row vector.

[0182] combining all target candidate words to obtain the compressed text.

[0183] In the embodiments of the present application, the obtained original text is segmented to obtain at least one text segmentation result, a combination matrix of each text segmentation result is obtained, the combination matrix is determined according to at least one of a segmentation matrix, a segmentation feature matrix and a segmentation position matrix, each combination matrix is compressed by using a plurality of preset compression matrices to obtain a target matrix, the target matrix corresponding to each combination matrix is merged to obtain a merged matrix, a target candidate word of each row vector of the merged matrix is determined from a preset dictionary, and all target candidate words are merged to obtain a compressed text. By performing multi-dimensional representation on the original text, the accuracy of understanding core knowledge points of the text can be improved, and then the accuracy, compression rate and meaning integrity of the original text content of the text compression can be improved.

[0184] The above describes the method provided by the present application. The device provided by the present application is described below.

[0185] Please refer to Figure 5 , a structural schematic diagram of a text compression device provided by an embodiment of the present application.

[0186] As shown in 56, the device can include:

[0187] The segmentation unit 510 is configured to segment the obtained original text to obtain at least one text segmentation result.

[0188] The acquisition unit 520 is configured to acquire a combination matrix of each text segmentation result, wherein the combination matrix is determined according to at least one of a segmentation matrix, a segmentation feature matrix and a segmentation position matrix.

[0189] The compression unit 530 is configured to compress each combination matrix by using a plurality of preset compression matrices to obtain a target matrix, wherein each compression matrix corresponds to a word attribute.

[0190] The merging unit 540 is configured to merge the target matrix corresponding to each combination matrix to obtain a merged matrix.

[0191] The determination unit 550 is configured to, for each row vector in the merged matrix, determine a probability of each word in a preset dictionary as a candidate word through the row vector, and take a candidate word with a probability meeting a requirement as a target candidate word of the row vector.

[0192] The output unit 560 is configured to merge all target candidate words to obtain a compressed text.

[0193] In some embodiments, the acquisition unit 520 is specifically configured to:

[0194] For each word segmentation in the text word segmentation result, a root vector of the word segmentation is determined according to semantic information of the word segmentation;

[0195] A part-of-speech vector of the word segmentation is determined according to part-of-speech information of the word segmentation;

[0196] A word position vector of the word segmentation is determined according to position information of the word segmentation in the text word segmentation result;

[0197] The root vector, the part-of-speech vector and the word position vector of all word segmentations in the text word segmentation result are combined respectively to obtain a word segmentation matrix, a word segmentation feature matrix and a word segmentation position matrix of the text word segmentation result.

[0198] In some embodiments, the obtaining unit 520 is specifically configured to:

[0199] The word segmentation matrix, the word segmentation feature matrix and the word segmentation position matrix are linearly transformed respectively by using corresponding first weight matrices and first bias terms, the first weight matrices and the first bias terms being obtained by pre-training;

[0200] The results of the linear transformation are summed to obtain the combination matrix.

[0201] In some embodiments, the compression unit 530 is specifically configured to:

[0202] For each combination matrix, each compression matrix and a displacement matrix corresponding to the compression matrix are used to compress the combination matrix;

[0203] The results of the compression processing are merged to obtain a target matrix corresponding to the combination matrix.

[0204] In some embodiments, the compression unit 530 is further configured to:

[0205] Similarities between the combination matrix and the target matrix are determined;

[0206] A first correlation degree between the original text vector and the combination matrix is determined, and a second correlation degree between the original text vector and the target matrix is determined;

[0207] A correlation difference between the first correlation degree and the second correlation degree is determined;

[0208] In a case where the similarity is greater than a first preset threshold and the correlation difference is less than a second preset threshold, the target matrix is compressed again by using the plurality of preset compression matrices to update the target matrix.

[0209] In some embodiments, the merging unit 540 is specifically configured to:

[0210] respectively, by using a corresponding second weight matrix and a second bias term, the second weight matrix and the second bias term being obtained by pre-training;

[0211] combining the results of the linear transformation to obtain the merging matrix.

[0212] In some embodiments, the determining unit 550 is specifically configured to:

[0213] obtaining vector distances between the row vector and word vectors corresponding to each word in the preset dictionary;

[0214] normalizing all the obtained vector distances to obtain probabilities of each word in the preset dictionary as a candidate word.

[0215] The implementation process of the functions and roles of each unit in the above apparatus is specifically described in the implementation process of the corresponding steps in the above method, which will not be repeated here.

[0216] The embodiments of the present application also provide a hardware structure. Referring to Figure 6 , Figure 6 the electronic device structure diagram provided by the embodiments of the present application. As shown in the figure, the hardware structure can include a processor and a machine readable storage medium, the machine readable storage medium stores machine executable instructions that can be executed by the processor; the processor is configured to execute the machine executable instructions to implement the method disclosed in the above examples of the present application. Figure 6

[0217] Based on the same application concept as the above method, the embodiments of the present application also provide a machine readable storage medium, the machine readable storage medium stores a plurality of computer instructions, and the computer instructions are executed by a processor to implement the method disclosed in the above examples of the present application.

[0218] For example, the machine readable storage medium can be: RAM (Random Access Memory, Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard disk drive), solid state disk, any type of storage disk (such as optical disk, DVD, etc.), or similar storage medium, or combination thereof.

[0219] ​It is to be understood that the phraseology or terminology such as "including" or "comprising" or the like used in the specification is taken to mean the instance of comprising, including, encompassing, containing or abstaining therefrom, and not to the exclusion of additional or other elements, or process steps. It is also to be understood that the use of relational terms such as "first", "second", "top" and / or "bottom", and the like are used solely to distinguish one from another entity or step without necessarily implying a specific order or sequence.

[0220] The above description is embodied only to explain the preferred embodiments of the present application, and is not used to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the present application.

Claims

1. A method of text compression, characterized by, The method comprises: segmenting the obtained original text to obtain at least one text segmentation result; obtaining a combination matrix of each text segmentation result, wherein the combination matrix is determined according to at least one of a segmentation matrix, a segmentation feature matrix and a segmentation position matrix; performing compression processing on each combination matrix by using a plurality of preset compression matrices to obtain a target matrix, wherein each compression matrix corresponds to a word attribute; merging the target matrix corresponding to each combination matrix to obtain a merged matrix; for each row vector in the merged matrix, determining the probability of each word in a preset dictionary as a candidate word through the row vector, and taking the candidate word with a probability meeting a requirement as a target candidate word of the row vector; merging all target candidate words to obtain a compressed text.

2. The method of claim 1, wherein, The method further comprises: for each segmentation in each text segmentation result, determining a root vector of the segmentation according to semantic information of the segmentation; determining a part-of-speech vector of the segmentation according to part-of-speech information of the segmentation; determining a word position vector of the segmentation according to position information of the segmentation in the text segmentation result; combining the root vector, the part-of-speech vector and the word position vector of all segmentations in the text segmentation result to obtain a segmentation matrix, a segmentation feature matrix and a segmentation position matrix of the text segmentation result.

3. The method of claim 1, wherein, The combination matrix is determined according to the segmentation matrix, the segmentation feature matrix and the segmentation position matrix, comprising: performing linear transformation on the segmentation matrix, the segmentation feature matrix and the segmentation position matrix respectively by using a corresponding first weight matrix and a first bias term, wherein the first weight matrix and the first bias term are obtained through pre-training; summing the results of the linear transformation to obtain the combination matrix.

4. The method of claim 1, wherein, The compression processing on each combination matrix by using a plurality of preset compression matrices to obtain a target matrix, comprising: for each combination matrix, performing compression processing on the combination matrix by using each compression matrix and a displacement matrix corresponding to the compression matrix; merging the results of the compression processing to obtain the target matrix corresponding to the combination matrix.

5. The method of claim 1, wherein, After the compression processing on each combination matrix by using a plurality of preset compression matrices to obtain a target matrix, the method further comprises: determining the similarity between the combination matrix and the target matrix; determining a first correlation degree between an original text vector and the combination matrix, and determining a second correlation degree between the original text vector and the target matrix; determining a correlation degree difference between the first correlation degree and the second correlation degree; in a case where the similarity is greater than a first preset threshold and the correlation degree difference is less than a second preset threshold, performing compression processing on the target matrix again by using the plurality of preset compression matrices to update the target matrix.

6. The method of claim 1, wherein, The merging of the target matrix corresponding to each combination matrix to obtain a merged matrix, comprising: performing linear transformation on the target matrix corresponding to each combination matrix by using a corresponding second weight matrix and a second bias term, wherein the second weight matrix and the second bias term are obtained through pre-training. The results of the linear transformation are combined to obtain the combination matrix.

7. The method of claim 1, wherein, The probability that each word in the preset dictionary is a candidate word is determined by the row vector, and the probability that each word in the preset dictionary is a candidate word is determined by the row vector. The vector distance between the row vector and the word vector corresponding to each word in the preset dictionary is obtained. All vector distances are normalized to obtain the probability that each word in the preset dictionary is a candidate word.

8. A text compression apparatus characterized by comprising: The device comprises: The segmentation unit is configured to segment the obtained original text to obtain at least one text segmentation result. The acquisition unit is configured to acquire a combination matrix of each text segmentation result, wherein the combination matrix is determined according to at least one of the segmentation matrix, the segmentation feature matrix, and the segmentation position matrix. The compression unit is configured to compress each combination matrix using a plurality of preset compression matrices to obtain a target matrix, wherein each compression matrix corresponds to a vocabulary attribute. The merging unit is configured to merge the target matrix corresponding to each combination matrix to obtain a combination matrix. The determination unit is configured to determine, for each row vector in the combination matrix, the probability that each word in the preset dictionary is a candidate word by the row vector, and to select a candidate word whose probability meets the requirement as a target candidate word of the row vector. The output unit is configured to merge all target candidate words to obtain a compressed text.

9. An electronic device, comprising: The processor and the memory are included, and the memory stores machine executable instructions that can be executed by the processor.

10. A machine-readable storage medium, characterized in that, The machine readable storage medium stores machine executable instructions, and the machine executable instructions are executed by the processor to implement the method in any one of claims 1-7.

11. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instructions are executed by the processor to implement the steps of the method in any one of claims 1-7.

Images