Text database updating method and device, equipment and storage medium

Abstract

The invention relates to a text database updating method and device, equipment and a storage medium. The method comprises the following steps: when first text data is detected by sliding a time window, calling second text data of a text temporary database, and respectively preprocessing the first text data and the second text data to obtain a first text preprocessing result and a second text preprocessing result; and determining a semantic relationship between the first text data and the second text data according to the two preprocessing results. And merging the first text data and the second text data according to the semantic relationship to obtain temporary updated text data. And after storing the temporary update text data in the text temporary database, calculating the data similarity between the temporary update text data and the cluster description data of each text cluster in the text historical database. And updating the temporary update text data corresponding to the maximum value of the data similarity to the text historical database. By adopting the method, the text database updating efficiency can be improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of text data processing, and in particular relates to a text database updating method, device, equipment and storage medium. Background Art

[0002] In many scenarios, it's necessary to update real-time text data into various text databases. Currently, newly generated text data is saved to a new location in the text database based on its timestamp. Duplicate data can be removed by deduplicating the text database. This approach is inefficient for scenarios involving multiple text databases and large amounts of data. Summary of the Invention

[0003] Based on this, it is necessary to provide a text database updating method, device, equipment and storage medium that can improve the text database updating efficiency in response to the above technical problems.

[0004] A text database updating method, the method comprising:

[0005] When the first text data is detected through the sliding time window, the second text data of the text temporary database is retrieved, and the first text data and the second text data are preprocessed respectively to obtain a first text preprocessing result and a second text preprocessing result.

[0006] A semantic relationship between the first text data and the second text data is determined according to the first text preprocessing result and the second text preprocessing result.

[0007] The first text data and the second text data are merged according to the semantic relationship to obtain temporary updated text data.

[0008] After the temporary updated text data is saved in the text temporary database, the data similarity between the temporary updated text data and the cluster description data of each text cluster in the text history database is calculated.

[0009] The temporary updated text data corresponding to the maximum data similarity value is updated to the corresponding text cluster in the text history database.

[0010] A text database updating device, comprising:

[0011] The preprocessing module is used to retrieve the second text data from the text temporary database when the first text data is detected through the sliding time window, and preprocess the first text data and the second text data respectively to obtain the first text preprocessing result and the second text preprocessing result.

[0012] The semantic relationship acquisition module is used to determine the semantic relationship between the first text data and the second text data according to the first text preprocessing result and the second text preprocessing result.

[0013] The merging module is used to merge the first text data and the second text data according to a semantic relationship to obtain temporary updated text data.

[0014] The similarity optimization module is used to calculate the data similarity between the temporary updated text data and the cluster description data of each text cluster in the text history database after saving the temporary updated text data in the text temporary database.

[0015] The updating module is used to update the temporary updated text data corresponding to the maximum data similarity value into the corresponding text cluster in the text history database.

[0016] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0017] When the first text data is detected through the sliding time window, the second text data of the text temporary database is retrieved, and the first text data and the second text data are preprocessed respectively to obtain a first text preprocessing result and a second text preprocessing result.

[0018] A semantic relationship between the first text data and the second text data is determined according to the first text preprocessing result and the second text preprocessing result.

[0019] The first text data and the second text data are merged according to the semantic relationship to obtain temporary updated text data.

[0020] After the temporary updated text data is saved in the text temporary database, the data similarity between the temporary updated text data and the cluster description data of each text cluster in the text history database is calculated.

[0021] The temporary updated text data corresponding to the maximum data similarity value is updated to the corresponding text cluster in the text history database.

[0022] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0023] When the first text data is detected through the sliding time window, the second text data of the text temporary database is retrieved, and the first text data and the second text data are preprocessed respectively to obtain a first text preprocessing result and a second text preprocessing result.

[0024] A semantic relationship between the first text data and the second text data is determined according to the first text preprocessing result and the second text preprocessing result.

[0025] The first text data and the second text data are merged according to the semantic relationship to obtain temporary updated text data.

[0026] After the temporary updated text data is saved in the text temporary database, the data similarity between the temporary updated text data and the cluster description data of each text cluster in the text history database is calculated.

[0027] The temporary updated text data corresponding to the maximum data similarity value is updated to the corresponding text cluster in the text history database.

[0028] The above-mentioned text database update method, device, equipment and storage medium firstly uses a sliding time window mechanism to capture the first text data in real time and synchronously retrieves the second text data in the temporary database to form a data processing batch, thereby avoiding frequent access to the main database. Secondly, through preprocessing technology, the dual data sources are subjected to noise reduction, word segmentation, entity recognition and other operations, key semantic features are extracted, and the logical relationship between the two is determined by combining the semantic similarity algorithm to achieve intelligent merging of text content, effectively reducing redundant data and reducing subsequent storage pressure. Then, the merged temporary update data is stored in the temporary database, and the predefined text cluster structure in the historical database (such as clustering based on dimensions such as topic, time, and entity) can be used to calculate the matching degree between the temporary data and the feature vectors of each cluster through algorithms such as cosine similarity and TF-IDF, so as to quickly locate the most relevant target cluster. Finally, an incremental update strategy is adopted to merge only the temporary data with the highest similarity into the corresponding cluster, avoiding full database traversal and repeated calculations, while maintaining the timeliness of the cluster structure through dynamic maintenance of cluster features. This solution achieves efficient integration of multi-source data by breaking down the data processing process into three parallel stages: real-time capture, semantic integration, and cluster matching. This solution utilizes a temporary database as a buffer layer. Through cluster feature indexing and a dynamic update mechanism, the computational complexity of data matching is significantly reduced, ensuring millisecond-level response times even in large-scale data scenarios. Furthermore, the layered architecture design provides the system with excellent scalability, enabling flexible response to the simultaneous updates of multiple databases, ultimately improving the overall efficiency of text database updates. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A flowchart of a text database updating method according to an embodiment;

[0030] Figure 2 Schematic diagram of a flow chart of a text cluster generation method in one embodiment;

[0031] Figure 3It is a structural block diagram of a text database updating device in one embodiment;

[0032] Figure 4 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0034] In one embodiment, Figure 1 As shown, a text database updating method is provided, comprising the following steps:

[0035] Step 102 : when the first text data is detected through the sliding time window, the second text data in the text temporary database is retrieved, and the first text data and the second text data are preprocessed respectively to obtain a first text preprocessing result and a second text preprocessing result.

[0036] Newly generated text data is recorded as first text data. A temporary text database stores text data within a preset time range. The text data in the temporary text database is recorded as second text data, and the temporary text database contains multiple pieces of second text data. A historical text database stores all text data generated in the past. The text data in the historical text database is recorded as third text data, and the historical text database contains multiple pieces of third text data. The third text data in the historical text database is clustered to obtain multiple text clusters on different topics, wherein each text cluster includes multiple pieces of third text data.

[0037] Text data can include news, hot topics, articles, blogs, e-books, comments, and emails. The following text uses news as an example. The first text data element is the latest news. The temporary text database stores today's news, while the historical text database stores news from the past few years. A text cluster is about a news topic, which can be categorized by hot events (e.g., news about a war between two countries) or content categories (e.g., news about economic development).

[0038] Preprocessing includes: word segmentation, stop word removal, part-of-speech tagging and named entity recognition.

[0039] Specifically, preprocess the first text data and the second text data respectively to obtain the first text preprocessing result and the second text preprocessing result. Perform word segmentation on the first text data and the second text data respectively to obtain the first text word set and the second text word set. Among them, both the first text word set and the second text word set contain multiple words; determine the union and intersection of the first text word set and the second text word set; calculate the data overlap based on the union and intersection.

[0040] Further, the word segmentation process is to split the sentences in the first text data and the second text data into words. The union of the first text word set and the second text word set includes all the words in the first text word set and the second text word set. The intersection of the first text word set and the second text word set is the common words of the first text word set and the second text word set. The ratio of the number of words in the intersection to the number of words in the union can be used as the data overlap.

[0041] The data overlap c between the first text data and the second text data can be calculated according to the following formula:

[0042]

[0043] Among them, c is the data overlap, I is the number of words in the intersection of the text word sets, and L is the number of words in the union of the text word sets.

[0044] Further, removing stop words is to remove common words such as "de", "shi", "zai" in the first text data and the second text data that contribute little to semantics.词性标注是识别第一文本数据与第二文本数据中每个词的词性(名词、动词、形容词等)。命名实体识别是识别第一文本数据与第二文本数据中人物、地点等名词。为了更清楚进行说明,假设第一文本数据与第二文本数据均为一条新闻,那么根据第一文本预处理结果和第二文本预处理结果确定该两条新闻之间的语义关系,语义关系包括该两条新闻是否有因果、转折、并列、递进等逻辑关系以及该两条新闻是否有时间上的先后顺序。最后依据语义关系将该两条新闻合并。比如若该两条新闻的内容是并列的,则可以使用“以及”、“和”等词语将该两条新闻连接到一起,若该两条新闻的内容是转折的,则可以使用“但是”、“却”等词语将该两条新闻连接到一起。(The content about "词性标注是识别第一文本数据与第二文本数据中每个词的词性(名词、动词、形容词等)。命名实体识别是识别第一文本数据与第二文本数据中人物、地点等名词。" seems to be incomplete or incorrect in the original Chinese. It's translated as it is for now.)To illustrate more clearly, assume that both the first text data and the second text data are a news item. Then, determine the semantic relationship between the two news items according to the first text preprocessing result and the second text preprocessing result. The semantic relationship includes whether there are logical relationships such as causality, turning, parallelism, and progression between the two news items, as well as whether there is a chronological order between the two news items. Finally, merge the two news items according to the semantic relationship. For example, if the content of the two news items is parallel, words such as "and", "as well as" can be used to connect the two news items. If the content of the two news items is turning, words such as "but", "however" can be used to connect the two news items.

[0045] Step 104, determine the semantic relationship between the first text data and the second text data according to the first text preprocessing result and the second text preprocessing result.

[0046] Step 106 : Merge the first text data and the second text data according to the semantic relationship to obtain temporary updated text data.

[0047] Specifically, the data overlap between the latest news and each piece of news stored in the text temporary database is calculated. If the data overlap between the latest news and a piece of news stored in the text temporary database is the largest and the data overlap is greater than or equal to the first threshold, the latest news is merged with the piece of news and saved in the text temporary database. After the merger, the updated content of the news is displayed (including the number of news, changes in news descriptions, and updates to news expression vectors). When the data overlap between the latest news and the piece of news is greater than or equal to the second threshold but less than the first threshold, the latest news is saved in the text temporary database, and the association between the latest news and the piece of news is added to the text temporary database. When the data overlap between the latest news and the piece of news is less than the second threshold, only the latest news is saved in the text temporary database. The update of the text temporary database is completed through the above-mentioned technical means.

[0048] Furthermore, the contextual relationship between adjacent sentences in the first text data is determined based on the first text preprocessing result, and each sentence in the first text data is divided based on the contextual relationship between adjacent sentences in the first text data to obtain multiple first segments, and the segment preprocessing result corresponding to each first segment is determined from the first text preprocessing result; the contextual relationship between adjacent sentences in the second text data is determined based on the second text preprocessing result, and each sentence in the second text data is divided based on the contextual relationship between adjacent sentences in the second text data to obtain multiple second segments, and the segment preprocessing result corresponding to each second segment is determined from the second text preprocessing result; for any first segment and any second segment: the semantic relationship between the first segment and the second segment is determined based on the segment preprocessing result corresponding to the first segment and the segment preprocessing result corresponding to the second segment, and the first segment is merged with the second segment based on the semantic relationship between the first segment and the second segment.

[0049] Step 108 : After the temporary updated text data is saved in the text temporary database, the data similarity between the temporary updated text data and the cluster description data of each text cluster in the text history database is calculated.

[0050] Specifically, the data similarity between each news item in the temporary text database and the cluster description data of each text cluster in the historical text database is calculated. If the data similarity between a news item in the temporary text database and the cluster description data of a text cluster is the highest, the news item is merged into the text cluster in the historical text database. After the merger, the updated content of the text cluster (including the number of text items in the text cluster, changes in the text cluster description, and updates to the text cluster expression vector) is displayed, completing the update of the historical text database.

[0051] Furthermore, based on the data similarities corresponding to the multiple text data with associated relationships in the text temporary database, the data similarities corresponding to the multiple text data are optimized; and each text data in the text temporary database is updated to the text cluster in the text history database with the greatest data similarity to the corresponding text data.

[0052] Furthermore, the association relationship between the first text data and the second text data includes multiple levels of strength (the greater the data overlap, the stronger the association relationship), for example, including three levels: strong, medium and weak. The strength level of the association relationship affects the weight of optimizing data similarity. Assuming that the association relationship is strong, the corresponding weight is 0.7, the association relationship is medium, and the corresponding weight is 0.5, and the association relationship is weak, the corresponding weight is 0.3. If the association relationship between the two text data in the text temporary database is strong, the data similarity between the first text data and the first text cluster in the text history database is the largest, with a value of 0.8, and the data similarity between the second text data and the first text cluster in the text history database is 0.6, then the optimized data similarity between the first text data and the first text cluster is equal to 0.8+0.6*0.7. The data similarity can be cosine similarity.

[0053] Step 110 : Take the temporary updated text data corresponding to the maximum data similarity value and update it into the corresponding text cluster in the text history database.

[0054] By updating the temporary text database and the historical text database, news data can be managed. Finally, news can be recommended to users based on the news in the temporary text database, and older news related to the news recommended to users can be determined based on the news in the historical text database. News within the same text cluster is related.

[0055] In the above-mentioned text database update method, first, a sliding time window mechanism is used to capture the first text data in real time, and the second text data in the temporary database is simultaneously retrieved, forming a data processing batch, thus avoiding frequent access to the main database. Secondly, preprocessing techniques are used to perform noise reduction, word segmentation, entity recognition, and other operations on the dual data sources, extracting key semantic features. Combined with a semantic similarity algorithm, the logical relationship between the two is determined, achieving intelligent merging of text content, effectively reducing redundant data and lowering subsequent storage pressure. Next, the merged temporary update data is stored in the temporary database. Using the predefined text cluster structure in the historical database (such as clustering based on dimensions such as topic, time, and entity), the matching degree between the temporary data and each cluster feature vector is calculated using algorithms such as cosine similarity and TF-IDF, quickly locating the most relevant target cluster. Finally, an incremental update strategy is adopted to merge only the most similar temporary data into the corresponding cluster, avoiding full database traversal and repeated calculations. At the same time, the timeliness of the cluster structure is maintained through dynamic maintenance of cluster features. This solution achieves efficient integration of multi-source data by breaking down the data processing process into three parallel stages: real-time capture, semantic integration, and cluster matching. This solution utilizes a temporary database as a buffer layer. Through cluster feature indexing and a dynamic update mechanism, the computational complexity of data matching is significantly reduced, ensuring millisecond-level response times even in large-scale data scenarios. Furthermore, the layered architecture design provides the system with excellent scalability, enabling flexible response to the simultaneous updates of multiple databases, ultimately improving the overall efficiency of text database updates.

[0056] In one embodiment, when first text data is detected through a sliding time window, second text data within a preset time range stored in a text temporary database is retrieved, and the first text data and the second text data are preprocessed respectively, specifically: sentences in the first text data and the second text data are segmented into words to obtain a first text word set and a second text word set. Non-semantic words in the first text word set and the second text word set are removed respectively, and the part of speech of each word in the first text word set and the second text word set is marked, and entity nouns in the first text word set and the second text word set are identified to obtain first text preprocessing results and second text preprocessing results.

[0057] In one embodiment, a first contextual relationship between adjacent sentences in the first text data is determined based on the first text preprocessing result, and each sentence in the first text data is divided according to the first contextual relationship to obtain a first segment preprocessing result. Furthermore, a second contextual relationship between adjacent sentences in the second text data is determined based on the second text preprocessing result, and each sentence in the second text data is divided according to the second contextual relationship to obtain a second segment preprocessing result.

[0058] It is worth noting that the contextual relationship between adjacent sentences is whether there is a correlation between adjacent sentences or the strength of the correlation. Assume that the first text data and the second text data are both news and contain multiple sentences. According to the first text preprocessing result, it is determined that the first, second and third sentences in the first text data are descriptions of the same object, then the first, second and third sentences in the first text data are divided into a first segment. Alternatively, according to the first text preprocessing result, it is determined that the similarity of the first, second and third sentences in the first text data is greater than the preset similarity (the correlation is judged to be strong), then the first, second and third sentences in the first text data are divided into a first segment.

[0059] Determining the segment preprocessing results corresponding to each first segment from the first text preprocessing results means dividing the first text preprocessing results according to each first segment to obtain the segment preprocessing results corresponding to each first segment. The division of the second segment is similar to the division of the first segment. Determine the semantic relationship between the first segment and the second segment based on the segment preprocessing results corresponding to the first segment and the segment preprocessing results corresponding to the second segment, including whether there is a semantic relationship and whether the semantic relationship is a logical relationship or a temporal sequence. Merging the first segment and the second segment based on the semantic relationship between the first segment and the second segment includes whether to merge and if so, according to which semantic relationship to merge (if there is no semantic relationship between the first segment and the second segment, then they are not merged).

[0060] In one embodiment, a text word union and a text word collection are obtained based on the first text word set and the second text word set, and a ratio of the number of words in the text word union to the number of words in the text word collection is calculated to obtain a data overlap:

[0061]

[0062] Wherein, c is the data overlap, I is the number of words in the text word set, and L is the number of words in the text word set. Segment semantic similarity is obtained based on any first segment in the first segment preprocessing result and any second segment in the second segment preprocessing result. After merging the first and second text data based on the inclusion relationship between the data overlap and a preset word overlap interval, the merged first and second text data are updated based on the judgment result between the segment semantic similarity and a preset segment association threshold to obtain temporarily updated text data.

[0063] In one embodiment, the maximum value of the word overlap interval is a first threshold, and the minimum value is a second threshold.

[0064] In one embodiment, when the data overlap is greater than or equal to a first threshold, the first text data and the second text data are merged to obtain temporary updated text data. When the data overlap is greater than or equal to a second threshold and less than the first threshold, the first text data and the semantic relationship between the first and second text data are used as temporary updated text data. When the data overlap is less than the second threshold, the first text data is used as temporary updated text data.

[0065] In one embodiment, a text history database includes multiple third text data. The Euclidean distance between any two third text data is calculated, and the Euclidean distance between the current third text data and the next third text data different from the current third text data is used as an edge to construct a complete text graph corresponding to the text history database. A minimum text tree is extracted from the complete text graph, wherein the minimum text tree is a connection method that includes all nodes in the complete text graph, and the sum of the edges in the minimum text tree is the smallest among all connection methods. The minimum text data is split into two text clusters by removing the longest edge in the minimum text data. After the two split text clusters are placed in a text cluster set, the following loop is executed: determine whether the number of nodes in all text clusters in the text cluster set of the current loop round is less than a preset number. If the judgment result is that all are less than, the text cluster in the text cluster set of the current loop round is used as the target text cluster, and the loop ends. Otherwise, the next loop is executed, and the longest edge in the text cluster in the text cluster set of the current loop round whose number of nodes is not less than the preset number is removed until the number of nodes in all text clusters in the text cluster set is less than the preset number.

[0066] It is worth noting that the text history database contains multiple third text data; the Euclidean distance between any two third text data in the text history database is calculated; each third text data is used as a node, and the Euclidean distance between any two third text data is used as an edge between the two third text data to construct a complete text graph corresponding to the text history database; a minimum text tree is extracted from the complete text graph, wherein the minimum text tree is a connection method that contains all nodes in the complete text graph, and the sum of the edges in the minimum text tree is the smallest among all connection methods; the minimum text tree is split into multiple text clusters by gradually removing the longest edges in the minimum text tree; and a summary of each text cluster is extracted as cluster description data of the text cluster.

[0067] For example, if there are 100 third text data in total, then the 100 third text data are regarded as 100 nodes, and the Euclidean distance between any two third text data is regarded as the edge between the two third text data to construct a complete text graph corresponding to the text history database. Next, remove the redundant edges in the complete text graph. As long as all the nodes in the complete text graph are included, a variety of connection methods can be obtained. Among the multiple connection methods, the connection method with the smallest sum of edges is selected as the minimum text tree. Finally, the longest edges in the minimum text tree are gradually removed to split the minimum text tree into multiple text clusters.

[0068] Furthermore, the longest edge in the minimum text tree is gradually removed to split the minimum text tree into multiple text clusters, including: removing the longest edge in the minimum text tree to split the minimum text tree into two text clusters; putting the two text clusters obtained by splitting into a text cluster set, and executing the following loop: judging whether the number of nodes in all text clusters in the text cluster set is less than a preset number; when the number of nodes in all text clusters in the text cluster set is less than the preset number, taking the text cluster in the text cluster set as the final text cluster and ending the loop; otherwise, removing the longest edge in the text cluster whose number of nodes in the text cluster set is not less than the preset number to split the text cluster into two text clusters.

[0069] Because the sum of the edges in the minimum text tree is the smallest among all connection methods, the minimum text tree is already the simplest connection method. Therefore, the minimum text tree has no redundant edges. As long as one edge is removed, the minimum text tree can be split into two text clusters. Determine the longest edge in the minimum text tree and remove it. At this time, the minimum text tree is split into two text clusters. The two text clusters obtained by splitting are placed into a text cluster set, and the following loop is executed: Continue to remove the longest edge in each text cluster in the text cluster set to split the text cluster into two text clusters until the number of nodes in all text clusters in the text cluster set is less than the preset number. The text cluster in the text cluster set is used as the final text cluster, and the loop ends. It should be noted that in a loop round, if the number of nodes in a text cluster is less than the preset number, then the text cluster will no longer be split in subsequent loop rounds.

[0070] In one embodiment, after the temporary updated text data is saved in the text temporary database, the data similarity between the temporary updated text data and the cluster description data of each text cluster in the text history database is optimized using a weight ratio according to the semantic relationship:

[0071]

[0072] Among them, w AB is the optimized data similarity between text data A and text data B, Xi is the embedded feature vector of text data A, Y i is the embedded feature vector of text data B, and n is the feature vector dimension.

[0073] In one embodiment, Figure 2 As shown, a text cluster generation method is provided, and the specific steps are as follows:

[0074] S201, calculating the Euclidean distance between any two third text data in the text history database;

[0075] S202: constructing a complete text graph corresponding to the text history database by taking each third text data as a node and the Euclidean distance between any two third text data as an edge between the two third text data;

[0076] S203, extracting a minimum text tree from the complete text graph, wherein the minimum text tree is a connection method that includes all nodes in the complete text graph, and the sum of the edges in the minimum text tree is the smallest among all connection methods;

[0077] S204, removing the longest edge in the minimum text tree to split the minimum text tree into two text clusters;

[0078] S205: Put the two text clusters obtained by splitting into a text cluster set and execute the following loop:

[0079] S206, determining whether the number of nodes in all text clusters in the text cluster set is less than a preset number;

[0080] S207, removing the target edge in the text cluster whose number of nodes is not less than a preset number in the text cluster set, so as to split the text cluster into two text clusters;

[0081] S208, when the number of nodes in all text clusters in the text cluster set is less than a preset number, the text cluster in the text cluster set is used as the final text cluster, and the loop ends;

[0082] S209: extracting the summary of each text cluster as cluster description data of the text cluster.

[0083] It is worth noting that a large language model can be used to extract summaries of each text cluster.

[0084] It should be understood that although Figure 1-2 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1-2At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0085] In one embodiment, Figure 3 As shown, a text database updating device is provided, comprising: a pre-processing module 302, a semantic relationship acquisition module 304, a merging module 306, a similarity optimization module 308 and an updating module 310, wherein:

[0086] The preprocessing module 302 is used to retrieve the second text data from the text temporary database when the first text data is detected through the sliding time window, and preprocess the first text data and the second text data respectively to obtain a first text preprocessing result and a second text preprocessing result.

[0087] The semantic relationship acquisition module 304 is configured to determine the semantic relationship between the first text data and the second text data according to the first text preprocessing result and the second text preprocessing result.

[0088] The merging module 306 is configured to merge the first text data and the second text data according to a semantic relationship to obtain temporarily updated text data.

[0089] The similarity optimization module 308 is configured to calculate the data similarity between the temporarily updated text data and the cluster description data of each text cluster in the text history database after saving the temporarily updated text data in the text temporary database.

[0090] The updating module 310 is configured to update the temporary updated text data corresponding to the maximum data similarity value into the corresponding text cluster in the text history database.

[0091] The specific definition of a text database updating device can be found in the definition of a text database updating method described above and will not be further elaborated here. Each module in the above-described text database updating device can be implemented in whole or in part via software, hardware, or a combination thereof. Each of these modules can be embedded in or independent of a processor in a computer device in hardware form, or can be stored in a memory in the computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0092] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 4As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a text database updating method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0093] Those skilled in the art will understand that Figure 3-4 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present invention and does not constitute a limitation on the computer device to which the solution of the present invention is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0094] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0095] When the first text data is detected through the sliding time window, the second text data of the text temporary database is retrieved, and the first text data and the second text data are preprocessed respectively to obtain a first text preprocessing result and a second text preprocessing result.

[0096] A semantic relationship between the first text data and the second text data is determined according to the first text preprocessing result and the second text preprocessing result.

[0097] The first text data and the second text data are merged according to the semantic relationship to obtain temporary updated text data.

[0098] After the temporary updated text data is saved in the text temporary database, the data similarity between the temporary updated text data and the cluster description data of each text cluster in the text history database is calculated.

[0099] The temporary updated text data corresponding to the maximum data similarity value is updated to the corresponding text cluster in the text history database.

[0100] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0101] When the first text data is detected through the sliding time window, the second text data of the text temporary database is retrieved, and the first text data and the second text data are preprocessed respectively to obtain a first text preprocessing result and a second text preprocessing result.

[0102] A semantic relationship between the first text data and the second text data is determined according to the first text preprocessing result and the second text preprocessing result.

[0103] The first text data and the second text data are merged according to the semantic relationship to obtain temporary updated text data.

[0104] After the temporary updated text data is saved in the text temporary database, the data similarity between the temporary updated text data and the cluster description data of each text cluster in the text history database is calculated.

[0105] The temporary updated text data corresponding to the maximum data similarity value is updated to the corresponding text cluster in the text history database.

[0106] Those skilled in the art will appreciate that all or part of the processes in the above-described embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the above-described embodiments. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAM bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM).

[0107] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0108] The above-described embodiments merely represent several implementation methods of the present invention. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent. It should be noted that a person skilled in the art would be able to make various modifications and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the patent for this invention shall be determined by the appended claims.

Claims

1. A text database updating method, characterized in that: The method comprises: When first text data is detected through the sliding time window, second text data is retrieved from the text temporary database, and the first text data and the second text data are preprocessed respectively to obtain a first text preprocessing result and a second text preprocessing result; determining a semantic relationship between the first text data and the second text data according to the first text preprocessing result and the second text preprocessing result; Merging the first text data and the second text data according to the semantic relationship to obtain temporary updated text data; After the temporary updated text data is saved in the text temporary database, the data similarity between the temporary updated text data and the cluster description data of each text cluster in the text history database is calculated; The temporary updated text data corresponding to the maximum data similarity value is updated into the corresponding text cluster in the text history database.

2. The method according to claim 1, characterized in that include: When first text data is detected through the sliding time window, second text data in the text temporary database is retrieved, and the first text data and the second text data are preprocessed respectively to obtain a first text preprocessing result and a second text preprocessing result, including: When the first text data is detected through the sliding time window, the second text data within the preset time range saved in the text temporary database is retrieved, and the first text data and the second text data are preprocessed respectively, specifically: the sentences in the first text data and the second text data are segmented into words respectively to obtain a first text word set and a second text word set; non-semantic words in the first text word set and the second text word set are removed respectively, and the part of speech of each word in the first text word set and the second text word set is marked, and the entity nouns of the first text word set and the second text word set are identified to obtain a first text preprocessing result and a second text preprocessing result.

3. The method according to claim 2, characterized in that Determining a semantic relationship between the first text data and the second text data according to the first text preprocessing result and the second text preprocessing result includes: determining a first contextual relationship between adjacent sentences in the first text data according to the first text preprocessing result, and dividing each sentence in the first text data according to the first contextual relationship to obtain a first segment preprocessing result; And determining a second contextual relationship between adjacent sentences in the second text data according to the second text preprocessing result, dividing each sentence in the second text data according to the second contextual relationship, and obtaining a second segment preprocessing result.

4. The method according to claim 3, characterized in that Merging the first text data and the second text data according to the semantic relationship to obtain temporarily updated text data includes: A text word union and a text word collection are obtained according to the first text word set and the second text word set, and the ratio of the number of words in the text word union to the number of words in the text word collection is calculated to obtain the data overlap: Wherein, c is the data overlap, I is the number of words in the text word set, and L is the number of words in the text word set; Obtaining segment semantic similarity according to any first segment in the first segment preprocessing result and any second segment in the second segment preprocessing result; After merging the first text data and the second text data according to the inclusion relationship between the data overlap degree and the preset word overlap interval, the merged first text data and the second text data are updated according to the judgment result between the segment semantic similarity and the preset segment association threshold to obtain temporarily updated text data.

5. The method according to claim 4, characterized in that The maximum value of the word overlap interval is a first threshold value, and the minimum value is a second threshold value; Merging the first text data and the second text data according to the inclusion relationship between the data overlap degree and a preset word overlap interval includes: When the data overlap degree is greater than or equal to a first threshold, merging the first text data with the second text data to obtain temporarily updated text data; When the data overlap is greater than or equal to a second threshold and less than the first threshold, the first text data and the semantic relationship between the first text data and the second text data are used as temporary update text data; When the data overlap is less than the second threshold, the first text data is used as temporary update text data.

6. The method according to claim 5, characterized in that The text history database includes a plurality of third text data; Before the step of calculating the data similarity between the temporary updated text data and the cluster description data of each text cluster in the text history database, the method further includes: Calculating the Euclidean distance between any two third text data, using the current third text data as a node and the Euclidean distance between the next third text data different from the current third text data as an edge, to construct a complete text graph corresponding to the text history database; Extracting a minimum text tree from the complete text graph, wherein the minimum text tree is a connection method that includes all nodes in the complete text graph, and the sum of edges in the minimum text tree is the smallest among all connection methods; The minimum text data is split into two text clusters by removing the longest edge in the minimum text data. After the two split text clusters are placed in a text cluster set, the following loop is executed: determine whether the number of nodes in all text clusters in the text cluster set of the current loop round is less than a preset number. If the judgment result is that they are all less than, the text cluster in the text cluster set of the current loop round is used as the target text cluster and the loop is ended; otherwise, the next loop is executed to remove the longest edge in the text cluster whose number of nodes in the text cluster set of the current loop round is not less than the preset number, until the number of nodes in all text clusters in the text cluster set is less than the preset number.

7. The method according to claim 6, characterized in that After the temporary updated text data is saved in the text temporary database, the data similarity between the temporary updated text data and the cluster description data of each text cluster in the text history database is calculated, including: After the temporary updated text data is saved in the text temporary database, the data similarity between the temporary updated text data and the cluster description data of each text cluster in the text history database is optimized using a weight ratio according to the semantic relationship: Among them, w AB is the optimized data similarity between text data A and text data B, X i is the embedded feature vector of text data A, Y i is the embedded feature vector of text data B, and n is the feature vector dimension.

8. A text database updating device, characterized in that: The device comprises: a preprocessing module configured to retrieve second text data from a temporary text database when first text data is detected through a sliding time window, and preprocess the first text data and the second text data respectively to obtain a first text preprocessing result and a second text preprocessing result; a semantic relationship acquisition module, configured to determine a semantic relationship between the first text data and the second text data according to the first text preprocessing result and the second text preprocessing result; a merging module, configured to merge the first text data and the second text data according to the semantic relationship to obtain temporarily updated text data; a similarity optimization module, configured to calculate the data similarity between the temporarily updated text data and the cluster description data of each text cluster in the text history database after saving the temporarily updated text data in the text temporary database; An updating module is used to update the temporary updated text data corresponding to the maximum data similarity value into the corresponding text cluster in the text history database.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images