Text processing method and device, electronic device and computer-readable storage medium

Abstract

The present application provides a text processing method and device, an electronic device, and a computer-readable storage medium. The method includes: processing multiple text data to generate multiple text vectors; constructing a text relationship graph according to the similarity between multiple text vectors; for a node in the text relationship graph, calculating the feature vector of the node according to the node and the path between the node and other nodes; calculating the weight value of each node according to the feature vector of each node, and selecting the text data corresponding to at least one node as the target text data according to the weight value. The embodiment of the present application determines the representative content text by the topological relationship between the content texts in the form of nodes and the scoring and ranking method, which can fully explore the correlation relationship between the content texts, thereby greatly improving the probability and accuracy of determining the representative content text.

Description

Technical Field

[0001] The present application relates to the field of text processing, and in particular to a text processing method and device, an electronic device, and a computer-readable storage medium. Background Art

[0002] As social media users continue to grow, a considerable number of news events have begun to spread through social media such as Weibo and Twitter. On social media, a lot of information can be generated for the same event. In order to facilitate users to better browse an event, one or several representative information will be selected from the thousands of information about the event and displayed as summary information.

[0003] In the prior art, summary information is usually selected based on factors such as release time, number of reposts or comments, word frequency, etc. However, information released earlier may not fully describe the event; if a user with high influence reposts a piece of information, the information selected based on the number of reposts or comments may not be representative; and the word importance calculated based on word frequency is also affected by the noise contained in the information, so the information selected in this way cannot guarantee its representativeness. Summary of the invention

[0004] The embodiments of the present application provide a text processing method and device, an electronic device, and a computer-readable storage medium to solve the defect that the selected summary information in the prior art is not accurate enough.

[0005] To achieve the above object, the present application provides a text processing method, including:

[0006] Processing multiple text data to generate multiple text vectors;

[0007] Constructing a text relationship graph according to the similarities between the multiple text vectors, wherein the text relationship graph includes: nodes composed of text vectors, and paths between text vectors composed of similarities between text vectors;

[0008] For a node in the text relationship graph, a feature vector of the node is calculated according to the node and the path between the node and other nodes;

[0009] According to the feature vector of each node, a weight value of each node is calculated, and according to the weight value, text data corresponding to at least one node is selected as target text data.

[0010] The present application also provides a text processing device, including:

[0011] A text vector generation module, used for processing a plurality of text data to generate a plurality of text vectors;

[0012] A construction module, used to construct a text relationship graph according to the similarities between the multiple text vectors, wherein the text relationship graph includes: nodes composed of text vectors, and paths between text vectors composed of similarities between text vectors;

[0013] A first calculation module is used to calculate a feature vector of a node in the text relationship graph according to the node and a path between the node and other nodes;

[0014] The second calculation module is used to calculate the weight value of each node according to the feature vector of each node;

[0015] The selection module is used to select text data corresponding to at least one node as target text data according to the weight value.

[0016] The present application also provides an electronic device, including:

[0017] Memory, used to store programs;

[0018] The processor is used to run the program stored in the memory to execute the text processing method provided in the embodiment of the present application.

[0019] An embodiment of the present application also provides a computer-readable storage medium on which a computer program executable by a processor is stored, wherein the program, when executed by the processor, implements the text processing method provided in the embodiment of the present application.

[0020] The text processing method and device, electronic device, and computer-readable storage medium provided in the embodiments of the present application can fully explore the association relationship between content texts by determining representative content texts based on the topological relationship between content texts in node form and scoring and ranking, thereby greatly improving the probability and accuracy of determining representative content texts.

[0021] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0023] Figure 1 A schematic diagram of a scenario of a text processing system according to an embodiment of the present application;

[0024] Figure 2 A flowchart of a text processing method according to an embodiment of the present application;

[0025] Figure 3 is a flowchart of a text processing method according to another embodiment of the present application;

[0026] Figure 4 is a schematic diagram of a text processing device according to one embodiment of the present application; and

[0027] Figure 5 It is a schematic structural diagram of an electronic device according to one embodiment of the present application. DETAILED DESCRIPTION

[0028] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0029] With the development of Internet technology, more and more people choose to publish various texts on the Internet, which has also created an era of information explosion. In the current situation of a large amount of information explosion on the Internet, there are often a large number of texts for a certain event, especially each text may only represent a user's one-sided view or a certain aspect of the interpretation of the event. Therefore, for users who pay attention to this event, they can only understand the situation of the event by reading various texts about this event, which may require the user to read a large amount of content text in a short time. Therefore, in the prior art, it is usually selected as a representative text for a certain event by selecting texts forwarded or commented on by many people on the Internet. In other words, in the prior art, the text forwarded or commented on by the user is regarded as the text concerned by the user. Therefore, the prior art regards the text concerned by the most people as the most representative content text for the event, so as to recommend it to other users to understand the event. However, in actual applications, the content text concerned by the most users is not necessarily an objective and representative text, and may often only contain eye-catching content or content that just meets the concerns of users of a specific group. Therefore, such content text is not very helpful for users who do not know the event to understand the cause of the matter.

[0030] In addition, in the prior art, there is also a method of recommending the earliest published content text for a certain event to users as a representative text. Such a scheme will rely on the writing style of the user who first published the content text about a certain event. For example, if the user who first discovered a certain event likes to summarize and organize the event, then the text published by such a user can be a representative content text, but if the user who first published the event simply disclosed the event without an in-depth or detailed introduction, then the content text published by such a user based solely on the early or late publishing time cannot actually play a representative role.

[0031] In this regard, the prior art has used computer technology to perform various statistical processing on the content texts on the Internet, so as to select the content texts as representative texts and recommend them to users based on the results of the statistical processing. Such a scheme is more objective than the above-mentioned scheme of simply using some indicators to screen the content texts. For example, the prior art can calculate the importance of words or character strings in each content text for a certain event. For example, the frequency of occurrence of the same words in each content text on the Internet, that is, the word frequency (TF), is counted, and the content text containing the maximum word frequency is regarded as a representative text, but the word frequency (TF) scheme may count commonly used auxiliary words or predicate words (for example, is) and the like as the words with the maximum word frequency, which will affect the accuracy of selecting content texts. Therefore, the prior art has used TF-IDF (word frequency-inverse document frequency) to remove the influence of meaningless words by applying different weights to words. Such a scheme is more effective for scientific research documents with rigorous word usage, and can be conducive to selecting keywords for documents. However, for a large number of content texts on the Internet, the words contained are varied, and Internet users may use different words of various styles to describe the same event. Therefore, words with higher frequency do not necessarily mean that the corresponding texts are highly representative.

[0032] Therefore, in this application, each content text is directly used as the processing object, and the relationship between the content texts is calculated, so as to construct a relationship graph of the content texts. In this relationship graph, each content text is a node, and the connection between the texts represents the relevance or similarity between the content texts. Therefore, a feature vector of the node can be generated based on each node in the relationship graph and all the connections of the node, and a score is calculated based on the feature vector, so as to select a representative feature vector, i.e., a content text, according to the score sorting.

[0033] Figure 1 FIG. 1 is a schematic diagram of a text processing system according to an embodiment of the present application. Figure 1 As shown in , multiple users can publish content texts for different events through the Internet. Figure 1 As shown in , there may be various content texts about events 1 and 2 published by multiple users on the Internet, among which some texts are published for event 1 and can be called original texts, and some texts may be content texts further published for other people's content texts on event 1, and therefore can also be called comment texts. In this application, whether it is content text published for an event or comment text published for content already published by others, they are collectively referred to as content texts. Due to the development of Internet technology, when it is necessary to quickly understand the tendency of event 1 or 2 or the general opinions of users, in the embodiments of this application, as shown in FIG. Figure 1As shown in, when various content texts published on the Internet are obtained, content texts for different events, such as event 1 and event 2, may be obtained. Therefore, the obtained content texts can be converted into text vectors first, and then these content texts can be classified or screened first, so that the content texts for the same event are classified together, and a relationship topology diagram of these text vectors is established, so that each text vector is represented by the nodes in the diagram and the similarity or correlation between the text vectors is represented by the lines between the nodes. Alternatively, all the content texts obtained can be placed in the same relationship topology diagram without classification and screening, regardless of which event it is for, and each text vector is represented by the nodes in the diagram and the relationship between the text vectors is represented by the lines between the nodes. In this case, the correlation or similarity between the content texts for different events will be lower, and will be displayed as being far away or completely without correlation in the relationship topology diagram. Therefore, the step of pre-processing can be omitted.

[0034] In the embodiment of the present application, a feature vector containing the content of the node and the relevance of the node to related nodes, i.e., the connection, can be generated for the nodes and the connection between the nodes in the relationship topology diagram. Therefore, by scoring such feature vectors, the relevance of each content text to the event under discussion or its importance to the event can be obtained, and by sorting according to the score, the text that can represent the event can be selected from the sorted sequence as a summary.

[0035] Therefore, by determining representative content texts based on the topological relationship between content texts in the form of nodes and scoring and ranking, the association relationship between content texts can be fully explored, thereby greatly improving the probability and accuracy of determining representative content texts.

[0036] Next reference Figures 2 to 5 A text processing method, a text processing device, an electronic device, and a computer-readable storage medium according to embodiments of the present application are described in detail.

[0037] Figure 2 FIG. 1 is a flow chart of a text processing method according to an embodiment of the present application. Figure 2 As shown in, according to an embodiment of the present application, the text processing method of the present application may include:

[0038] S201, processing a plurality of text data to generate a plurality of text vectors.

[0039] In the embodiment of the present application, various text data related to the event can be obtained from various data sources, for example, various text data related to the event can be obtained from a database of an Internet forum, a database of a social software, or uploaded by a user. In the embodiment of the present application, the obtained text data can be ensured to be text data related to the event by selecting a database as a source for obtaining text data.

[0040] After the text data is acquired, in step S201, a process such as word embedding can be used for each text data to generate a corresponding text vector according to each text data. For example, when generating a text vector, various vector generation methods in the art can be used, such as LSTM models. Specifically, each text data can be first segmented and a word vector for each word can be generated through word embedding, and then the word vector can be input into, for example, an LSTM model to generate a text vector. In the process, the attention mechanism can also be further used to calculate the importance, i.e., the weight, of each word vector of the text data in the process of generating the text vector, so that the generated text vector contains more information about relatively important words.

[0041] In step S201 , the above process may be performed on each of a plurality of text data, so that a plurality of text vectors may be generated.

[0042] S202: construct a text relationship graph according to the similarities between the multiple text vectors.

[0043] After generating text vectors for the acquired multiple text data in step S201, in step S202, a text relationship graph can be constructed for these text vectors. For example, a relationship graph of text vectors can be constructed in the form of a topological relationship graph. The function of such a topological relationship graph is to calculate and display the correlation, i.e., the relationship, between each text vector. Specifically, the text relationship graph may include: nodes composed of text vectors, and paths between text vectors composed of the similarities between text vectors. For example, for a certain text, the similarity between the text vector and other text vectors can be calculated as the correlation between text vectors, and the text vector can be connected to other text vectors based on the similarity. For example, a node of a text vector with a higher similarity to the calculated text vector can have a shorter distance from the node of the calculated text vector. Therefore, by forming nodes with text vectors and forming the lines between the nodes in the relationship graph with the correlation between text vectors, that is, the paths between text vectors in the text relationship graph are formed by the similarity between text vectors. In particular, in step S202, if the similarity calculated for two text vectors is lower than a predetermined threshold, for example, 0.2, it can be considered that there is no correlation between the two text vectors, and in the text relationship graph, no connecting line is formed between the nodes of the two text vectors, that is, no path relationship between the nodes of the two text vectors is constructed in step S202.

[0044] In addition, when calculating the similarity between text vectors in step S202, the similarity between nodes can be calculated using, for example, cosine similarity (cosine), that is, the cosine value of the angle between the two text vectors can be used as the similarity between the two text vectors, thereby determining the connection between the nodes corresponding to the two text vectors, that is, the path.

[0045] S203: For a node in the text relationship graph, a feature vector of the node is calculated according to the node and the path between the node and other nodes.

[0046] In an embodiment of the present application, a feature vector of the node can be generated for a node in a text relationship graph and the path between the node and other nodes. As described above, the text relationship graph constructed in step S202 includes nodes composed of text vectors and the correlation between text vectors, such as the path between text vectors composed of similarities. Therefore, in step S203, a feature vector containing the text vector information of the node and the correlation between the node and other nodes, i.e., the relationship, can be generated for each node as the feature vector of the node. For example, GCN (Graph Convolutional Network) can be used to traverse each node in the relationship graph and each edge in the graph, so that a feature vector containing the relationship between the node and other nodes can be quickly generated for each node in the relationship graph.

[0047] For example, in an embodiment of the present application, a word vector is generated for each text data through word embedding processing, and a text vector containing the relationship between these word vectors is further generated based on these word vectors.

[0048] S204, calculating a weight value of each node according to the feature vector of each node, and selecting text data corresponding to at least one node as target text data according to the weight value.

[0049] After obtaining the feature vector of each node in step S203, the weight value of each node, that is, the importance of the node relative to other nodes, can be calculated. For example, since the feature vector of each node contains the text vector data corresponding to the node and the correlation with other texts that exceeds the predetermined threshold. For example, a node in the relationship diagram includes, in addition to its text content information, its correlation relationship with two other nodes, while another node includes, in addition to its text content information, its correlation relationship with eight other nodes. Therefore, it is obvious that in step S204, it can be calculated that the weight value of the node with correlation relationship with the other eight nodes is higher than the weight value of the node with correlation only with two other nodes.

[0050] After the weight values ​​are calculated, the weight values ​​of each node may be sorted. For example, a softmax algorithm may be used to sort the weight values. Therefore, at least one text data with the highest weight value may be selected from the multiple text data obtained in step S201 according to the weight values, for example, the content text of the node with the highest weight value may be selected as the most representative text.

[0051] The text processing method provided in the embodiment of the present application can fully explore the correlation relationship between content texts by determining representative content texts based on the topological relationship between content texts in node form and scoring and ranking, thereby greatly improving the probability and accuracy of determining representative content texts.

[0052] Figure 3 FIG. 1 is a flow chart of a text processing method according to another embodiment of the present application. Figure 3 As shown in, according to an embodiment of the present application, the text processing method of the present application may include:

[0053] S301, multiple text data are processed through the attention mechanism and the LSTM model to generate multiple text vectors.

[0054] In an embodiment of the present application, after acquiring the text data, in step S301, a process such as word embedding can be used for each text data to generate a corresponding text vector according to each text data. When generating a text vector, each text data can be first segmented and a word vector for each word can be generated by word embedding, and an attention mechanism can be used to calculate the importance of each word vector of the text data in the process of generating the text vector, that is, the weight, so that the word vector can be input into, for example, an LSTM model to generate a text vector. The introduction of the attention mechanism can make the generated text vector contain more information about relatively important words. In step S301, the above process can be performed for each of the multiple text data, so that multiple text vectors can be generated.

[0055] S302, calculating the cosine similarity between two text vectors in the plurality of text vectors as the similarity between the two text vectors.

[0056] S303: construct a text relationship graph according to the cosine similarities between the multiple text vectors.

[0057] After generating text vectors for the acquired multiple text data in step S301, in step S302, the similarity between the text vectors can be calculated using, for example, cosine similarity, that is, the cosine value of the angle between the two text vectors can be used as the similarity between the two text vectors.

[0058] In step S303, a text relationship graph can be constructed for these text vectors according to the cosine similarity calculated in step S302. For example, a relationship graph of text vectors can be constructed in the form of a topological relationship graph. The function of such a topological relationship graph is to calculate and display the correlation, i.e., the relationship, between each text vector. Specifically, the text relationship graph may include: nodes composed of text vectors, and paths between text vectors composed of similarities between text vectors. For example, for a certain text vector, the cosine value of the angle between the text vector and other text vectors can be calculated in step S302 as the similarity between text vectors, and the text vector is connected to other text vectors according to the similarity in step S303, thereby forming a text relationship graph. For example, a node of a text vector with a higher similarity to the calculated text vector may have a shorter distance from the node of the calculated text vector. Therefore, by forming nodes with text vectors and forming the lines between the nodes in the relationship graph with the correlation between the text vectors, that is, the paths between the text vectors in the text relationship graph are formed by the similarity between the text vectors. In particular, in step S302, if the similarity calculated for two text vectors is lower than a predetermined threshold, for example, 0.2, it can be considered that there is no correlation between the two text vectors, and when constructing the text relationship graph in step S303, no connecting line is formed between the nodes of the two text vectors, that is, the nodes of the two text vectors are not connected in step S303.

[0059] S304: For a node in the text relationship graph, a feature vector of the node is calculated according to the node and the path between the node and other nodes.

[0060] In an embodiment of the present application, a feature vector of the node can be generated for each node in the text relationship graph constructed in step S303 and the path relationship between the node and other nodes. As described above, the text relationship graph constructed in step S303 includes nodes composed of text vectors, and paths between text vectors composed of similarities between text vectors. Therefore, in step S304, a feature vector containing information about the node and the similarity between the node and other nodes can be generated for each node as the feature vector of the node.

[0061] For example, in step S304, GCN (Graph Convolutional Network) can be used to traverse each node in the relationship graph and each connection in the graph, so as to quickly generate a feature vector for each node in the text relationship graph that contains its relationship with other nodes.

[0062] S305 , calculating an average value vector according to the feature vectors of the multiple nodes to identify the events targeted by the multiple nodes.

[0063] After obtaining the feature vector of each node in step S304, the average vector of the feature vectors of all nodes can be calculated to represent the entire event involved in these nodes. In other words, in an embodiment of the present application, since the multiple text data obtained can be text data for a specific event, the content contained in each text data can be views or opinions given from different angles for the event or further discussions on the views given by other users. Therefore, by calculating the average value of the text vectors of all text data related to the event, a vector representation that reflects the overall view or average view of the entire event, that is, the average value vector, can be obtained.

[0064] S306: Generate a merged vector for each feature vector according to the average value vector and the multiple feature vectors.

[0065] After the average value vector of the events involved in the multiple nodes is obtained in step S305, the average value vector can be merged with the feature vector of each node obtained in step S304 in step S306 to generate a merged vector that not only contains the content information of each text data and the relationship between the text data and other text data, but also contains the overall description of the events involved. Therefore, by generating the merged vector, it can be regarded as adding the background information of the event to the feature vector obtained in step S304 that only reflects the single text data itself and the correlation with the related text data, so that the generated merged vector can contain more comprehensive information.

[0066] S307, calculating the score of the merged vector of each node.

[0067] In step S307, a score can be calculated for the merged vector of each text data, for example, the weight value of each merged vector, that is, the importance of the merged vector relative to other merged vectors, can be calculated. For example, since the merged vector of each node contains the data of the node and the relevance of the node exceeding a predetermined threshold and the overall information of the event. For example, a node in the relationship graph includes, in addition to its text content information, its correlation relationship with two other nodes, while another node includes, in addition to its text content information, its correlation relationship with eight other nodes. Therefore, it is obvious that in step S204, it can be calculated that the weight value of the node having a correlation relationship with the other eight nodes is higher than the weight value of the node having a correlation only with two other nodes.

[0068] In addition, in the embodiment of the present application, when calculating the score of the merged vector, the following formula can be used to calculate the score S: i :

[0069] Si =f(x i )=v T tanh(W s Y i +W c C)

[0070] Among them, v T , W s and W c All are parameters, Y i is the feature vector calculated in step S304, and C is the average value vector calculated in step S305.

[0071] In particular, the parameters used in the above calculations can be trained through a machine learning model. For example, the recognized target text data can be input as training samples into the machine learning model for training, thereby optimizing the parameters.

[0072] S308, sorting the multiple feature vectors according to the weight values ​​to form an ordered feature vector sequence.

[0073] After calculating the weight value, the weight value of each node can be sorted. For example, a softmax algorithm can be used to sort the weight values ​​to obtain an ordered feature vector sequence.

[0074] S309, selecting the text data of the node corresponding to the first feature vector located at the front end of the feature vector sequence after sorting as the target text data.

[0075] In step S309, at least one text data with the highest weight value may be selected from the feature vector sequence obtained after sorting in step S308. For example, the content text of the node with the highest weight value may be selected as the most representative text.

[0076] In addition, according to an embodiment of the present application, the operation of obtaining a summary of an event can also be applied in other scenarios, such as in the screening of product reviews, in the screening of answers in a question-and-answer community, in the screening of comments on video files, and so on, which are not limited here. The above video files can be movies, TV series, short videos, and so on. In this application scenario, a video file can be used as a target event, and the comments on the video file can be used as a text object describing the video file. In this way, according to the method of this embodiment, at least one comment can be screened out from multiple comments describing the video file, and displayed to the user as a summary describing the video file. In this way, the user can understand the main content of the video file based on the summary of the video file, so as to consider whether to watch the video file based on the summary.

[0077] The text processing method provided in the embodiment of the present application can fully explore the correlation relationship between content texts by determining representative content texts based on the topological relationship between content texts in node form and scoring and ranking, thereby greatly improving the probability and accuracy of determining representative content texts.

[0078] Figure 4 FIG. 1 is a schematic diagram of a text processing device according to an embodiment of the present application. Figure 2 and Figure 3 The method steps shown. Figure 4 As shown, the text processing device of the embodiment of the present application includes: a text vector generating module 401, a constructing module 402, a first calculating module 403, a second calculating module 404 and a selecting module 405.

[0079] The text vector generating module 401 may be used to process a plurality of text data to generate a plurality of text vectors.

[0080] In the embodiment of the present application, the text processing device can obtain various text data related to the event from various data sources, for example, various text data related to the event can be obtained from the database of the Internet forum, the database of the social software, or uploaded by the user. In the embodiment of the present application, the database as the source of the text data can be selected to ensure that the acquired text data is text data related to the event.

[0081] After acquiring the text data, the text vector generation module 401 of the text processing device of the present application can generate a corresponding text vector according to each text data using a process such as word embedding for each text data. For example, when generating a text vector, various vector generation methods in the art can be used, such as LSTM models, etc. For example, the text processing module 401 can first perform word segmentation processing on each text data and generate a word vector for each word through word embedding processing, and then the word vector can be input into, for example, an LSTM model to generate a text vector. In the process, the attention mechanism can also be further used to calculate the importance, i.e., weight, of each word vector of the text data in the process of generating the text vector, so that the generated text vector contains more information about relatively important words.

[0082] Therefore, the text vector generation module 401 of the present application can perform the above-mentioned processing on each of the multiple text data, thereby generating multiple text vectors.

[0083] The construction module 402 is used to construct a text relationship graph based on the similarities between multiple text vectors. After the text vector generation module 401 generates text vectors for the acquired multiple text data, the construction module 402 can construct a text relationship graph based on these text vectors. For example, the relationship graph of the text vectors can be constructed in the form of a topological relationship graph. The function of such a topological relationship graph is to calculate and display the correlation, i.e., the relationship, between each text vector. Specifically, the text relationship graph may include: nodes composed of text vectors, and paths between text vectors composed of the similarities between text vectors.

[0084] In an embodiment of the present application, the construction module 402 may further include a first calculation unit 4021 and a construction unit 4022. The first calculation unit 4021 may be used to calculate the similarity between text vectors using, for example, a cosine similarity (cosine). For example, the first calculation unit 4021 may use the cosine value of the angle between the two text vectors as the similarity between the two text vectors, thereby determining the connection between the nodes corresponding to the two text vectors. This construction unit 4022 may be used to construct a text relationship graph based on the cosine similarity between multiple text vectors. For example, for a certain text, the first calculation unit 4021 calculates the similarity between the text vector and other text vectors as the correlation between the text vectors, and the construction unit 4012 may connect the text vector to other text vectors based on the similarity calculated by the first calculation unit 4021. For example, a node of a text vector having a higher similarity to a text vector being calculated may have a shorter distance from a node of a text vector being calculated. Therefore, the construction unit 4022 can use the text vectors as nodes, and the correlation between the text vectors as the connection lines between the nodes in the relationship graph, that is, the similarity between each text vector and other text vectors is used as the path from the node to other nodes in the text relationship graph. In particular, if the similarity calculated by the first calculation unit 4021 for two text vectors is lower than a predetermined threshold, for example, 0.2, it can be considered that there is no correlation between the two text vectors, and when the construction unit 4022 constructs the text relationship graph, no connection line is formed between the nodes of the two text vectors, that is, the construction unit 4022 may not form a path relationship between the nodes of the two text vectors.

[0085] The first calculation module 403 may be used to calculate, for a node in the text relationship graph, a feature vector of the node according to the node and a path between the node and other nodes.

[0086] In an embodiment of the present application, the first computing module 403 can generate a feature vector of the node for each node in the text relationship graph constructed by the construction module 402 and its relationship with other nodes. As described above, in the text relationship graph constructed by the construction module 402, the text vector is represented as each node in the graph, and the lines between the nodes are used to represent the correlation between the text vectors, such as similarity. Therefore, the first computing module 403 can generate a feature vector containing the information of the node and the correlation between it and other nodes, that is, the relationship, for each node as the feature vector of the node. For example, GCN (Graph Convolutional Network) can be used to traverse each node in the relationship graph and each edge in the graph, so that a feature vector containing its relationship with other nodes can be quickly generated for each node in the relationship graph.

[0087] The second calculation module 404 may be configured to calculate a weight value of each node according to the feature vector of each node calculated by the first calculation module 403 .

[0088] The second calculation module 404 can calculate the weight value of each node according to the feature vector of each node calculated by the first calculation module 403, that is, the importance of the node relative to other nodes. For example, since the feature vector of each node contains the data of the node and the relevance of its relevance exceeding the predetermined threshold. For example, a node in the relationship diagram includes, in addition to its text content information, its correlation relationship with two other nodes, while another node includes, in addition to its text content information, its correlation relationship with eight other nodes. Therefore, it is obvious that the weight value of the node with correlation relationship with the other eight nodes can be calculated to be higher than the weight value of the node with correlation only with two other nodes.

[0089] In addition, the second calculation module 404 may further include a second calculation unit 4041 , a first vector generation unit 4042 , and a third calculation unit 4043 .

[0090] The second calculation unit 4041 may be configured to calculate an average value vector according to the feature vectors of the multiple nodes, so as to identify the events targeted by the multiple nodes.

[0091] After the first calculation module 403 obtains the feature vector of each node, the second calculation unit 4041 can calculate the average value vector of the feature vectors of all nodes to represent the entire event involved in these nodes. In other words, in an embodiment of the present application, since the multiple text data obtained can be text data for a specific event, the content contained in each text data can be views or opinions given from different angles for the event or further discussions on the views given by other users. Therefore, by calculating the average value of the text vectors of all text data related to the event, a vector representation of the overall view or average view of the entire event, that is, the average value vector, can be obtained.

[0092] The first vector generating unit 4042 may be configured to generate a merged vector for each feature vector according to the average value vector and the plurality of feature vectors.

[0093] After the second calculation unit 4041 obtains the average value vector of the events involved in the multiple nodes, the first vector generation unit 4042 can merge the average value vector with the feature vector of each node obtained by the first calculation module 403 to generate a merged vector that not only contains the content information of each text data itself and the relationship between the text data and other text data, but also contains the overall description of the events involved. Therefore, by generating the merged vector, it can be regarded as adding the background information of the event to the feature vector obtained by the first calculation module 403 that only reflects the single text data itself and the correlation with the related text data, so that the generated merged vector can contain more comprehensive information.

[0094] The third calculation unit 4043 can be used to calculate the score of the merged vector of each node. For example, the third calculation unit 4043 can calculate the weight value of each merged vector, that is, the importance of the merged vector relative to other merged vectors. For example, since the merged vector of each node contains the data of the text vector and the correlation whose correlation exceeds the predetermined threshold and the overall information of the event. For example, a node in the relationship graph includes, in addition to its text content information, its correlation relationship with other two nodes, while another node includes, in addition to its text content information, its correlation relationship with other eight nodes. Therefore, it is obvious that the third calculation unit 4043 can calculate that the weight value of the node having a correlation relationship with the other eight nodes is higher than the weight value of the node having a correlation only with the other two nodes.

[0095] In addition, in the embodiment of the present application, when calculating the score of the merged vector, the following formula can be used to calculate the score S: i :

[0096] S i =f(xi )=v T tanh(W s Y i +W c C)

[0097] Among them, v T , W s and W c All are parameters, Y i is the feature vector calculated by the first calculation module 403, and C is the average value vector calculated by the second calculation unit 4041.

[0098] In particular, the parameters used in the above calculations can be trained by a machine learning model. For example, the second calculation module 404 can further include a parameter acquisition unit 4044, which can be used to obtain parameters obtained by machine learning based on training data. Therefore, the third calculation unit 4043 can use the parameters obtained by the parameter acquisition unit 4044 to perform calculations.

[0099] The selection module 405 may be used to select text data corresponding to at least one node as target text data according to the weight value calculated by the second calculation module 404. For example, the selection module 405 may include a sorting unit 4051 and a selection unit 4052.

[0100] The sorting unit 4051 can be used to sort the multiple feature vectors according to the weight value to form an ordered feature vector sequence. For example, after the second calculation module 404 calculates the weight value, the sorting unit 4051 can sort the weight value of each node. For example, a softmax algorithm can be used to sort the weight value to obtain an ordered feature vector sequence.

[0101] The selection unit 4052 can select the text data of the node corresponding to the first feature vector at the front end of the feature vector sequence after sorting as the target text data. For example, at least one text data with the highest weight value can be selected from the feature vector sequence obtained after sorting by the sorting unit 4051, for example, the content text of the node with the highest weight value can be selected as the most representative text.

[0102] The text processing device provided in the embodiment of the present application can fully explore the correlation relationship between content texts by determining representative content texts based on the topological relationship between content texts in node form and scoring and ranking, thereby greatly improving the probability and accuracy of determining representative content texts.

[0103] The internal functions and structure of the text processing device are described above. The device can be implemented as an electronic device. Figure 5FIG. 1 is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Figure 5 As shown, the electronic device includes a memory 51 and a processor 52 .

[0104] The memory 51 is used to store programs. In addition to the above programs, the memory 51 can also be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device, contact data, phone book data, messages, pictures, videos, etc.

[0105] The memory 51 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0106] The processor 52 is not limited to a central processing unit (CPU), but may also be a processing chip such as a graphics processing unit (GPU), a field programmable gate array (FPGA), an embedded neural network processor (NPU) or an artificial intelligence (AI) chip. The processor 52 is coupled to the memory 51 and executes a program stored in the memory 51. When the program is run, any of the above-mentioned text processing methods is executed.

[0107] Further, if Figure 5 As shown, the electronic device may also include: a communication component 53, a power component 54, an audio component 55, a display 56 and other components. Figure 5 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 5 Components shown.

[0108] The communication component 53 is configured to facilitate wired or wireless communication between the electronic device and other devices. The electronic device can access a wireless network based on a communication standard, such as WiFi, 3G, 4G or 5G, or a combination thereof. In an exemplary embodiment, the communication component 53 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 53 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0109] The power supply component 54 provides power to various components of the electronic device. The power supply component 54 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device.

[0110] The audio component 55 is configured to output and / or input audio signals. For example, the audio component 75 includes a microphone (MIC), and when the electronic device is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 51 or sent via the communication component 53. In some embodiments, the audio component 55 also includes a speaker for outputting audio signals.

[0111] The display 56 includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0112] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A text processing method, characterized in that: include: Processing multiple text data to generate multiple text vectors; Constructing a text relationship graph according to the similarities between the multiple text vectors, wherein the text relationship graph includes: nodes composed of text vectors, and paths between text vectors composed of similarities between text vectors; For a node in the text relationship graph, a feature vector of the node is calculated according to the node and the path between the node and other nodes; The weight value of each node is calculated according to the feature vector of each node, and the weight value of each node is calculated according to the feature vector of each node, including: calculating an average value vector according to the feature vectors of multiple nodes to identify the events corresponding to the multiple nodes; generating a merge vector for each feature vector according to the average value vector and the multiple feature vectors; calculating the score of the merge vector of each node as the weight value, and selecting text data corresponding to at least one node as target text data according to the weight value, wherein the content contained in the text data includes views or opinions given from different angles on the event.

2. The text processing method according to claim 1, wherein: The step of constructing a text relationship graph according to the similarities between the multiple text vectors includes: Calculating the cosine similarity between two text vectors in the plurality of text vectors as the similarity between the two text vectors; Construct a text relationship graph based on the cosine similarity between multiple text vectors.

3. The text processing method according to claim 1, wherein: The processing of the plurality of text data to generate a plurality of text vectors includes: The multiple text data are processed through an attention mechanism and an LSTM model to generate the multiple text vectors.

4. The text processing method according to claim 1, wherein: The step of calculating the score of the merged vector of each node comprises: The score of the merge vector for each node is calculated using the parameters obtained through machine learning based on the training data.

5. The text processing method according to claim 1, wherein: The selecting text data corresponding to at least one text vector as target text data according to the weight value comprises: Sorting the plurality of feature vectors according to the weight values ​​to form an ordered feature vector sequence; The text data of the node corresponding to the first feature vector located at the front end of the feature vector sequence after sorting is selected as the target text data.

6. A text processing device, comprising: A text vector generation module, used for processing a plurality of text data to generate a plurality of text vectors; A construction module, used to construct a text relationship graph according to the similarities between the multiple text vectors, wherein the text relationship graph includes: nodes composed of text vectors, and paths between text vectors composed of similarities between text vectors; A first calculation module is used to calculate a feature vector of a node in the text relationship graph according to the node and a path between the node and other nodes; a second calculation module, used to calculate the weight value of each node according to the feature vector of each node; the second calculation module includes: a second calculation unit, used to calculate an average value vector according to the feature vectors of multiple nodes to identify the events targeted by the multiple nodes; a first vector generation unit, used to generate a merge vector for each feature vector according to the average value vector and the multiple feature vectors; a third calculation unit, used to calculate the score of the merge vector of each node; A selection module is used to select text data corresponding to at least one node as target text data according to the weight value; the content contained in the text data includes views or opinions given from different angles on the event.

7. The text processing device according to claim 6, wherein: The building blocks further include: A first calculation unit, configured to calculate a cosine similarity between two text vectors in the plurality of text vectors as a similarity between the two text vectors; A construction unit is used to construct a text relationship graph according to the multiple text vectors and the similarities between the text vectors calculated by the first calculation unit.

8. The text processing device according to claim 6, wherein: The text vector generation module is further used for: The multiple text data are processed through an attention mechanism and an LSTM model to generate the multiple text vectors.

9. The text processing device according to claim 6, wherein: The second calculation module further comprises: A parameter acquisition unit, used to acquire parameters for machine learning based on training data, and The third calculation unit calculates a score of the merge vector of each node using the parameters acquired by the parameter acquisition unit.

10. The text processing device according to claim 6, wherein: The selection module comprises: A sorting unit, used for sorting the plurality of feature vectors according to the weight values ​​to form an ordered feature vector sequence; The selection unit is used to select the text data of the node corresponding to the first feature vector at the front end of the feature vector sequence formed by the sorting unit as the target text data.

11. An electronic device, wherein: include: Memory, used to store programs; A processor, configured to run the program stored in the memory to execute the text processing method as claimed in any one of claims 1 to 5.

12. A computer-readable storage medium having stored thereon a computer program executable by a processor, wherein: When the program is executed by a processor, the text processing method as claimed in any one of claims 1 to 5 is implemented.

Images