Data updating method, device and storage medium

Abstract

The present application relates to the field of computer technology, and in particular to a method, device and storage medium for updating data. The method comprises: obtaining multiple text data from a database; screening out preliminary candidate data that meets preset conditions from the multiple text data; determining the semantic similarity between each preliminary candidate data and all text data in the database, and determining the text data with a semantic similarity greater than a preset value in all text data as final candidate data; determining the semantic repetition penalty value of each final candidate data according to the semantic similarity between the preliminary candidate data and the final candidate data; determining the score of each final candidate data according to the data attributes and the semantic repetition penalty value of each final candidate data; determining the data to be removed from all final candidate data according to the scores of all final candidate data; removing the data to be removed from the database can effectively reduce the amount of data calculation, improve calculation efficiency, and increase the update speed of the database.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device and storage medium for updating data. Background Art

[0002] The retrieval database of the existing RAG (Retrieval-Augmented Generation) system is formed by manually collecting various documents, retrieving API documents from the Internet, and then converting these data into structured data. Therefore, the data in the database of the existing RAG system is fixed and cannot be updated in real time, resulting in an increasing amount of data in the database, which may contain some erroneous, incomplete, outdated, and redundant data. In addition, there are often multiple data with the same meaning in the database, and the semantic similarity between multiple data is too large.

[0003] The RAG system needs to continuously obtain data from the database, but the huge and complex data in the existing database makes it impossible for the RAG system to obtain data in a timely and accurate manner. Therefore, it is urgent to update the data in the database. However, the existing data update technology often directly calculates all the data in the database, which leads to a large amount of duplicate data being repeatedly calculated and used, which not only increases the calculation cost and amount of calculation, but also the huge data calculation often makes the calculation results inaccurate, resulting in poor database data update effect and difficulty in data update. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a method, device and storage medium for data updating, so as to solve the problem of large amount of calculation and difficulty in updating data in the prior art.

[0005] In order to achieve the above-mentioned object, the first aspect of the present application provides a method for updating data, the method comprising:

[0006] Get multiple text data from the database;

[0007] Filter out preliminary candidate data that meet preset conditions from multiple text data;

[0008] Determine the semantic similarity between each preliminary candidate data and all text data in the database, and determine the text data with a semantic similarity greater than a preset value among all text data as final candidate data;

[0009] Determine a semantic duplication penalty value for each final candidate data according to the semantic similarity between the preliminary candidate data and the final candidate data;

[0010] Determine the score of each final candidate data according to the data attribute and the semantic duplication penalty value of each final candidate data;

[0011] Determine the data to be removed from all the final candidate data according to the scores of all the final candidate data;

[0012] Remove the data to be removed from the database.

[0013] In an embodiment of the present application, determining the semantic repetition penalty value of each final candidate data based on the semantic similarity between the preliminary candidate data and the final candidate data includes: for any final candidate data, determining the number of repetitions of the preliminary candidate data whose semantic similarity with the preliminary candidate data exceeds a threshold, and determining the semantic repetition penalty value of any final candidate data based on the number of repetitions.

[0014] In an embodiment of the present application, determining the score of each final candidate data based on the data attributes and semantic repetition penalty value of each final candidate data includes: when the semantic repetition penalty value of any final candidate data is not the initial value, obtaining at least one preliminary candidate data whose semantic similarity exceeds a threshold; comparing the initial scores corresponding to any final candidate data and at least one preliminary candidate data, the initial scores being determined by the data attributes of the respective data; when the initial score corresponding to any final candidate data is not less than the initial score corresponding to at least one preliminary candidate data, resetting the semantic repetition penalty value of any final candidate data to the initial value, and determining the score of any final candidate data based on the data attributes and semantic repetition penalty value of any final candidate data.

[0015] In an embodiment of the present application, the data attributes include user evaluation, existence time and number of uses of each preliminary candidate data, and the method also includes: for each preliminary candidate data, determining the like rate or dislike rate of the preliminary candidate data according to the user evaluation of the preliminary candidate data; for each preliminary candidate data, determining the time decay rate of the preliminary candidate data according to the existence time and decay coefficient of the preliminary candidate data; arranging all the preliminary candidate data in descending order based on the like rate, or arranging all the preliminary candidate data in ascending order based on the dislike rate to obtain the first ranking of each preliminary candidate data; arranging all the preliminary candidate data in ascending order based on the time decay rate to obtain the second ranking of each preliminary candidate data; arranging all the preliminary candidate data in descending order based on the number of uses to obtain the third ranking of each preliminary candidate data; determining the initial score of each preliminary candidate data according to the first ranking, second ranking and third ranking of each preliminary candidate data.

[0016] In an embodiment of the present application, determining the initial score of each preliminary candidate data according to the first ranking, second ranking and third ranking of each preliminary candidate data includes: obtaining a first weight coefficient representing the importance of user evaluation in the data attribute, a second weight coefficient representing the importance of existence duration in the data attribute, and a third weight coefficient representing the importance of number of uses in the data attribute; for each preliminary candidate data, determining a first ratio between the first weight coefficient of the preliminary candidate data and the sum of the first ranking and the sorting constant parameter, a second ratio between the second weight coefficient and the sum of the second ranking and the sorting constant parameter, and a third ratio between the third weight coefficient and the sum of the third ranking and the sorting constant parameter; and determining the sum of the first ratio, the second ratio and the third ratio of each preliminary candidate data as the initial score of each preliminary candidate data.

[0017] In an embodiment of the present application, screening out preliminary candidate data that meets preset conditions from multiple text data includes: obtaining data attributes of the multiple text data, wherein the data attributes include user evaluation, existence duration, and usage count of each text data; for any one of the multiple text data, if a click-through rate in the user evaluation of the text data is greater than a preset click-through rate, determining the text data to be preliminary candidate data; for any one of the multiple text data, if an existence duration of the text data is greater than a preset duration, determining the text data to be preliminary candidate data; for any one of the multiple text data, if a usage count of the text data is less than a preset count, determining the text data to be preliminary candidate data.

[0018] In an embodiment of the present application, determining the data to be removed from all the final candidate data according to the scores of all the final candidate data includes: arranging all the final candidate data in ascending order according to all the scores; and selecting a preset number of final candidate data at the end from all the arranged final candidate data as the data to be removed.

[0019] In an embodiment of the present application, the method also includes: after determining the data to be removed from all the final candidate data based on the scores of all the final candidate data, sending the data to be removed to the user, and re-storing the data to be removed that has been reviewed by the user into the database; backing up the data to be removed that has not been reviewed by the user, and removing the data to be removed that has not been reviewed by the user from the database.

[0020] A second aspect of the present application provides a data updating device, including:

[0021] a memory configured to store instructions;

[0022] The processor is configured to call instructions from the memory and implement the above-mentioned data updating method when executing the instructions.

[0023] A third aspect of the present application provides a machine-readable storage medium, on which instructions are stored, and the instructions are used to enable a machine to execute the above-mentioned data updating method.

[0024] Through the above technical scheme, preliminary candidate data that meet the preset conditions can be screened out from multiple text data in the database, and the semantic similarity between each preliminary candidate data and all the text data in the database can be determined, and then the text data with a semantic similarity greater than a preset value in all the text data can be determined as the final candidate data; the semantic repetition penalty value of each final candidate data is determined according to the semantic similarity between the preliminary candidate data and the final candidate data; the score of each final candidate data is determined according to the data attributes and the semantic repetition penalty value of each final candidate data; the data to be removed from all the final candidate data is determined according to the scores of all the final candidate data; removing the data to be removed from the database can effectively reduce the amount of data calculation, improve calculation efficiency, and increase the update speed of the database.

[0025] Other features and advantages of the embodiments of the present application will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present application, but do not constitute a limitation on the embodiments of the present application. In the accompanying drawings:

[0027] Figure 1 A schematic diagram of a process flow of a method for updating data according to an embodiment of the present application is schematically shown;

[0028] Figure 2 Another flowchart of a method for updating data according to an embodiment of the present application is schematically shown;

[0029] Figure 3 The internal structure diagram of a computer device according to an embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the specific implementation methods described herein are only used to illustrate and explain the embodiments of the present application, and are not used to limit the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0031] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present application, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0032] Figure 1 A schematic diagram of a data updating method according to an embodiment of the present application is shown. Figure 1 As shown, an embodiment of the present application provides a method for updating data, which may include the following steps.

[0033] Step 101: Acquire multiple text data from a database.

[0034] Step 102: Filter out preliminary candidate data that meet preset conditions from multiple text data.

[0035] The processor may obtain multiple text data from the database. After obtaining the multiple text data, the processor may filter out preliminary candidate data that meets preset conditions from the multiple text data.

[0036] In an embodiment of the present application, screening out preliminary candidate data that meets preset conditions from multiple text data includes: obtaining data attributes of the multiple text data, wherein the data attributes include user evaluation, existence duration, and usage count of each text data; for any one of the multiple text data, if a click-through rate in the user evaluation of the text data is greater than a preset click-through rate, determining the text data to be preliminary candidate data; for any one of the multiple text data, if an existence duration of the text data is greater than a preset duration, determining the text data to be preliminary candidate data; for any one of the multiple text data, if a usage count of the text data is less than a preset count, determining the text data to be preliminary candidate data.

[0037] The processor can filter out preliminary candidate data that meets preset conditions from multiple text data. Specifically, the processor can obtain data attributes of multiple text data, wherein the data attributes include user evaluation, existence time and number of uses of each text data. For any one of the multiple text data, the processor can determine whether the click-down rate in the user evaluation of the text data is greater than the preset click-down rate. The preset click-down rate can be set based on actual conditions. In the case where the click-down rate in the user evaluation of the text data is greater than the preset click-down rate, the processor can determine that the text data is preliminary candidate data. For any one of the multiple text data, the processor can determine whether the existence time of the text data is greater than the preset time. The preset time can be set based on actual conditions. In the case of determining that the existence time of the text data is greater than the preset time, the processor can determine that the text data is preliminary candidate data. For any one of the multiple text data, the processor can determine whether the number of uses of the text data is less than the preset number of times. The preset number of times can be determined based on actual conditions. When the number of times the text data is used is less than a preset number, the processor may determine that the text data is preliminary candidate data, so as to filter out the preliminary candidate data from the text data, reduce the amount of data that needs to be processed subsequently, and improve processing efficiency.

[0038] Step 103: Determine the semantic similarity between each preliminary candidate data and all text data in the database, and determine the text data with a semantic similarity greater than a preset value among all text data as the final candidate data.

[0039] Step 104: Determine a semantic duplication penalty value for each final candidate data according to the semantic similarity between the preliminary candidate data and the final candidate data.

[0040] After obtaining the preliminary candidate data, the processor can determine the semantic similarity between each preliminary candidate data and all text data in the database, and determine the text data with a semantic similarity greater than a preset value among all text data as the final candidate data. The preset value can be determined based on actual conditions. After determining the final candidate data, the processor can determine the semantic duplication penalty value of each final candidate data based on the semantic similarity between the preliminary candidate data and the final candidate data.

[0041] In an embodiment of the present application, determining the semantic repetition penalty value of each final candidate data based on the semantic similarity between the preliminary candidate data and the final candidate data includes: for any final candidate data, determining the number of repetitions of the preliminary candidate data whose semantic similarity with the preliminary candidate data exceeds a threshold, and determining the semantic repetition penalty value of any final candidate data based on the number of repetitions.

[0042] The processor can determine the semantic repetition penalty value of each final candidate data according to the semantic similarity between the preliminary candidate data and the final candidate data. Specifically, for any final candidate data, the processor can determine the number of repetitions of the preliminary candidate data whose semantic similarity with the preliminary candidate data exceeds a threshold. And determine the semantic repetition penalty value of any final candidate data according to the number of repetitions. The threshold value can be determined based on actual conditions. In an optional embodiment, the semantic repetition penalty value of any final candidate data can be equal to the number of repetitions.

[0043] Step 105: Determine the score of each final candidate data according to the data attribute and the semantic duplication penalty value of each final candidate data.

[0044] After obtaining the semantic duplication penalty value of each final candidate data, the processor may determine the score of each final candidate data according to the data attribute and the semantic duplication penalty value of each final candidate data.

[0045] In an embodiment of the present application, determining the score of each final candidate data based on the data attributes and semantic repetition penalty value of each final candidate data includes: when the semantic repetition penalty value of any final candidate data is not the initial value, obtaining at least one preliminary candidate data whose semantic similarity exceeds a threshold; comparing the initial scores corresponding to any final candidate data and at least one preliminary candidate data, the initial scores being determined by the data attributes of the respective data; when the initial score corresponding to any final candidate data is not less than the initial score corresponding to at least one preliminary candidate data, resetting the semantic repetition penalty value of any final candidate data to the initial value, and determining the score of any final candidate data based on the data attributes and semantic repetition penalty value of any final candidate data.

[0046] The processor can determine the score of each final candidate data according to the data attributes and semantic repetition penalty value of each final candidate data. Specifically, in the case where the semantic repetition penalty value of any final candidate data is not the initial value, the processor can obtain at least one preliminary candidate data whose semantic similarity with the any final candidate data exceeds a threshold. And compare the initial scores corresponding to any final candidate data and at least one preliminary candidate data, wherein the initial scores can be determined by the data attributes of the respective data. In the case where the initial score corresponding to any final candidate data is not less than the initial score corresponding to at least one preliminary candidate data, the processor can reset the semantic repetition penalty value of any final candidate data to the initial value, and determine the score of any final candidate data based on the data attributes and semantic repetition penalty value of any final candidate data, so as to avoid penalizing all data and ensure the validity of the data.

[0047] In an embodiment of the present application, the data attributes include user evaluation, existence time and number of uses of each preliminary candidate data, and the method also includes: for each preliminary candidate data, determining the like rate or dislike rate of the preliminary candidate data according to the user evaluation of the preliminary candidate data; for each preliminary candidate data, determining the time decay rate of the preliminary candidate data according to the existence time and decay coefficient of the preliminary candidate data; arranging all the preliminary candidate data in descending order based on the like rate, or arranging all the preliminary candidate data in ascending order based on the dislike rate to obtain the first ranking of each preliminary candidate data; arranging all the preliminary candidate data in ascending order based on the time decay rate to obtain the second ranking of each preliminary candidate data; arranging all the preliminary candidate data in descending order based on the number of uses to obtain the third ranking of each preliminary candidate data; determining the initial score of each preliminary candidate data according to the first ranking, second ranking and third ranking of each preliminary candidate data.

[0048] The data attributes include the user evaluation, existence time and number of times used of each preliminary candidate data. For each preliminary candidate data, the processor can determine the like rate or dislike rate of the preliminary candidate data according to the user evaluation of the preliminary candidate data. The user evaluation refers to the number of likes and dislikes of each preliminary candidate data by all users. The time decay rate of the preliminary candidate data is determined according to the existence time and decay coefficient of the preliminary candidate data, that is, the time decay rate w=e −λt , w is the time decay rate, e is the base of the natural logarithm function, λ is the decay coefficient, which can be determined based on the time half-life, and t is the existence duration. After obtaining the like rate or dislike rate of each preliminary candidate data, the processor can arrange them in descending order based on the like rate of all preliminary candidate data, or in ascending order based on the dislike rate of all preliminary candidate data to obtain the first ranking of each preliminary candidate data, that is, the first ranking represents the ranking of the like rate or dislike rate of each preliminary candidate data in the like rate or dislike rate of all preliminary candidate data. After obtaining the time decay rate of each preliminary candidate data, the processor can arrange them in ascending order based on the time decay rate of all preliminary candidate data to obtain the second ranking of each preliminary candidate data, that is, the second ranking represents the ranking of the time decay rate of each preliminary candidate data in the time decay rate of all preliminary candidate data. The processor can also arrange all preliminary candidate data in descending order based on the number of times they are used to obtain the third ranking of each preliminary candidate data, that is, the third ranking represents the ranking of the number of times each preliminary candidate data is used in the number of times all preliminary candidate data are used. After obtaining the first ranking, the second ranking, and the third ranking of each preliminary candidate data, the processor may determine an initial score for each preliminary candidate data.

[0049] In an embodiment of the present application, determining the initial score of each preliminary candidate data according to the first ranking, second ranking and third ranking of each preliminary candidate data includes: obtaining a first weight coefficient representing the importance of user evaluation in the data attribute, a second weight coefficient representing the importance of existence duration in the data attribute, and a third weight coefficient representing the importance of number of uses in the data attribute; for each preliminary candidate data, determining a first ratio between the first weight coefficient of the preliminary candidate data and the sum of the first ranking and the sorting constant parameter, a second ratio between the second weight coefficient and the sum of the second ranking and the sorting constant parameter, and a third ratio between the third weight coefficient and the sum of the third ranking and the sorting constant parameter; and determining the sum of the first ratio, the second ratio and the third ratio of each preliminary candidate data as the initial score of each preliminary candidate data.

[0050] The processor can determine the initial score of each preliminary candidate data according to the first ranking, the second ranking and the third ranking of each preliminary candidate data. Specifically, the processor can obtain the first weight coefficient α1 representing the importance of user evaluation in the data attribute, the second weight coefficient α2 representing the importance of the existence time in the data attribute, and the third weight coefficient α3 representing the importance of the number of times used in the data attribute. For each preliminary candidate data d, the processor determines the first ratio α1 / (k+rank1(d)) between the first weight coefficient α1 of the preliminary candidate data d and the sum of the first ranking rank1(d) and the sorting constant parameter k, the second ratio α2 / (k+rank2(d)) between the second weight coefficient α2 and the sum of the second ranking rank2(d) and the sorting constant parameter k, and the third ratio α3 / (k+rank3(d)) between the third weight coefficient α3 and the sum of the third ranking rank3(d) and the sorting constant parameter k. The sorting constant parameter can be determined based on actual conditions and is a constant. After obtaining the first ratio, the second ratio and the third ratio of each preliminary candidate data, the processor can determine the sum of the first ratio α1 / (k+ rank1(d)), the second ratio α2 / (k+ rank2(d)) and the third ratio α3 / (k+ rank3(d)) of each preliminary candidate data as the initial score of each preliminary candidate data, that is, the initial score of preliminary candidate data d is score=α1 / (k+ rank1(d))+α2 / (k+ rank2(d))+α3 / (k+ rank3(d)), so as to analyze each preliminary candidate data from all aspects and improve the comprehensiveness and accuracy of the initial score.

[0051] Step 106: Determine the data to be removed from all the final candidate data according to the scores of all the final candidate data.

[0052] Step 107: Remove the data to be removed from the database.

[0053] After obtaining the score of each candidate data, the processor may determine the data to be removed from all the final candidate data according to the scores of all the final candidate data, and remove the data to be removed from the database.

[0054] In an embodiment of the present application, determining the data to be removed from all the final candidate data according to the scores of all the final candidate data includes: arranging all the final candidate data in ascending order according to all the scores; and selecting a preset number of final candidate data at the end from all the arranged final candidate data as the data to be removed.

[0055] The processor may determine the data to be removed from all the final candidate data according to the scores of all the final candidate data. Specifically, the processor may arrange all the final candidate data in ascending order according to all the scores to obtain all the arranged final candidate data. And select a preset number of final candidate data at the rear from all the arranged final candidate data as the data to be removed. The preset number may be determined based on actual needs.

[0056] In an embodiment of the present application, the method also includes: after determining the data to be removed from all the final candidate data based on the scores of all the final candidate data, sending the data to be removed to the user, and re-storing the data to be removed that has been reviewed by the user into the database; backing up the data to be removed that has not been reviewed by the user, and removing the data to be removed that has not been reviewed by the user from the database.

[0057] After determining the data to be removed from all the final candidate data according to the scores of all the final candidate data, the processor can send the data to be removed to the user so that the user can review the data to be removed. The data to be removed that has been reviewed by the user is stored back in the database, and the data to be removed that has not been reviewed by the user is removed from the database, so as to update the database according to the user's needs.

[0058] In the embodiments of the present application, Figure 2As shown, the processor can obtain user feedback from the RAG system, and based on the user feedback, the positive feedback question and answer data is deduplicated in the form of QA (Question Answering System) and updated to the database. For example, the likes data can be updated to the database. For the data in the database, the processor can filter out a batch of preliminary candidate data from the database based on the data attributes, and the data attributes include user feedback, time, and number of uses. After filtering out a batch of preliminary candidate data, the processor can determine the semantic similarity between each preliminary candidate data and all the data in the database, and determine the data with a semantic similarity greater than a preset value as the final candidate data, and the preset value can be determined based on the actual situation. The processor can determine the repetition penalty item of each final candidate data based on the semantic similarity of the preliminary candidate data and the final candidate data funds. And the processor can determine the like (dislike) rate ranking of each final candidate data based on the user feedback in the data attributes, calculate the time decay rate ranking of each final candidate data based on the time in the data attributes, and calculate the data hit number ranking of each final candidate data based on the number of uses in the data attributes. After obtaining the likes (dislikes) rate ranking, time decay rate ranking, data hit count ranking, and duplicate penalty items of each final candidate data, the processor can perform a weighted RRF (Reciprocal rank fusion) score calculation based on the likes (dislikes) rate ranking, time decay rate ranking, data hit count ranking, and duplicate penalty items of each final candidate data to obtain the score of each final candidate data. After obtaining the score of each final candidate data, the processor can determine the data candidate to be removed based on the score of each final candidate data. And send the data candidate to be removed to the expert for review. The processor can update the data candidate to be removed that has passed the expert review to the data, and delete the data candidate to be removed that has failed the review. In an optional embodiment, the processor can remove the data after backing up the data candidate to be removed, so as to update the database efficiently.

[0059] The above technical solution can effectively reduce the amount of data calculation, improve computing efficiency, and increase the update speed of the database.

[0060] Figure 1 and Figure 2 FIG. 1 is a flow chart of a method for updating data in one embodiment. It should be understood that although Figure 1 and Figure 2 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 and Figure 2At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0061] The present application also provides a data updating device, including:

[0062] a memory configured to store instructions;

[0063] The processor is configured to call instructions from the memory and implement the above-mentioned data updating method when executing the instructions.

[0064] An embodiment of the present application also provides a machine-readable storage medium, on which instructions are stored, and the instructions are used to enable a machine to execute the above-mentioned data updating method.

[0065] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 3 As shown. The computer device includes a processor A01, a network interface A02, a memory (not shown in the figure) and a database (not shown in the figure) connected through a system bus. Among them, the processor A01 of the computer device is used to provide computing and control capabilities. The memory of the computer device includes an internal memory A03 and a non-volatile storage medium A04. The non-volatile storage medium A04 stores an operating system B01, a computer program B02 and a database (not shown in the figure). The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 in the non-volatile storage medium A04. The database of the computer device is used to store data such as text data, preliminary candidate data, semantic similarity, final candidate data, semantic repetition penalty values, scores, and data to be removed. The network interface A02 of the computer device is used to communicate with an external terminal through a network connection. When the computer program B02 is executed by the processor A01, a method for updating data is implemented.

[0066] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0067] An embodiment of the present application provides a device, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, the following steps are implemented: obtaining multiple text data from a database; screening out preliminary candidate data that meets preset conditions from the multiple text data; determining the semantic similarity between each preliminary candidate data and all text data in the database, and determining the text data with a semantic similarity greater than a preset value among all text data as final candidate data; determining a semantic repetition penalty value for each final candidate data based on the semantic similarity between the preliminary candidate data and the final candidate data; determining a score for each final candidate data based on data attributes and the semantic repetition penalty value of each final candidate data; determining data to be removed from all final candidate data based on the scores of all final candidate data; and removing the data to be removed from the database.

[0068] In one embodiment, determining the semantic repetition penalty value of each final candidate data based on the semantic similarity between the preliminary candidate data and the final candidate data includes: for any final candidate data, determining the number of repetitions between it and the preliminary candidate data whose semantic similarity in the preliminary candidate data exceeds a threshold, and determining the semantic repetition penalty value of any final candidate data based on the number of repetitions.

[0069] In one embodiment, determining the score of each final candidate data based on the data attributes and semantic repetition penalty value of each final candidate data includes: when the semantic repetition penalty value of any final candidate data is not the initial value, obtaining at least one preliminary candidate data whose semantic similarity exceeds a threshold; comparing the initial scores corresponding to any final candidate data with at least one preliminary candidate data, the initial scores being determined by the data attributes of the respective data; when the initial score corresponding to any final candidate data is not less than the initial score corresponding to at least one preliminary candidate data, resetting the semantic repetition penalty value of any final candidate data to the initial value, and determining the score of any final candidate data based on the data attributes and semantic repetition penalty value of any final candidate data.

[0070] In one embodiment, the data attributes include user evaluation, existence time and number of uses of each preliminary candidate data, and the method further includes: for each preliminary candidate data, determining the like rate or dislike rate of the preliminary candidate data according to the user evaluation of the preliminary candidate data; for each preliminary candidate data, determining the time decay rate of the preliminary candidate data according to the existence time and decay coefficient of the preliminary candidate data; arranging all the preliminary candidate data in descending order based on the like rate, or arranging all the preliminary candidate data in ascending order based on the dislike rate to obtain the first ranking of each preliminary candidate data; arranging all the preliminary candidate data in ascending order based on the time decay rate to obtain the second ranking of each preliminary candidate data; arranging all the preliminary candidate data in descending order based on the number of uses to obtain the third ranking of each preliminary candidate data; determining the initial score of each preliminary candidate data according to the first ranking, second ranking and third ranking of each preliminary candidate data.

[0071] In one embodiment, determining the initial score of each preliminary candidate data according to the first ranking, second ranking and third ranking of each preliminary candidate data includes: obtaining a first weight coefficient representing the importance of user evaluation in the data attribute, a second weight coefficient representing the importance of existence duration in the data attribute, and a third weight coefficient representing the importance of number of uses in the data attribute; for each preliminary candidate data, determining a first ratio between the first weight coefficient of the preliminary candidate data and the sum of the first ranking and the sorting constant parameter, a second ratio between the second weight coefficient and the sum of the second ranking and the sorting constant parameter, and a third ratio between the third weight coefficient and the sum of the third ranking and the sorting constant parameter; and determining the sum of the first ratio, the second ratio and the third ratio of each preliminary candidate data as the initial score of each preliminary candidate data.

[0072] In one embodiment, screening out preliminary candidate data satisfying preset conditions from a plurality of text data comprises: acquiring data attributes of the plurality of text data, wherein the data attributes comprise user evaluation, existence duration and number of uses of each text data; for any one of the plurality of text data, determining the text data as preliminary candidate data when a click-through rate in the user evaluation of the text data is greater than a preset click-through rate; for any one of the plurality of text data, determining the text data as preliminary candidate data when an existence duration of the text data is greater than a preset duration; for any one of the plurality of text data, determining the text data as preliminary candidate data when a number of uses of the text data is less than a preset number of times.

[0073] In one embodiment, determining the data to be removed from all the final candidate data according to the scores of all the final candidate data includes: arranging all the final candidate data in ascending order according to all the scores; and selecting a preset number of final candidate data at the end of all the arranged final candidate data as the data to be removed.

[0074] In one embodiment, the method also includes: after determining the data to be removed from all the final candidate data based on the scores of all the final candidate data, sending the data to be removed to the user, and re-storing the data to be removed that has been reviewed by the user into the database; backing up the data to be removed that has not been reviewed by the user, and removing the data to be removed that has not been reviewed by the user from the database.

[0075] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing the program of the method steps of initializing data updating.

[0076] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0077] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0078] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0079] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0080] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0081] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0082] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0083] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0084] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A method for updating data, characterized in that: The method comprises: Get multiple text data from the database; Filtering out preliminary candidate data satisfying preset conditions from the plurality of text data; Determine the semantic similarity between each preliminary candidate data and all text data in the database, and determine the text data with a semantic similarity greater than a preset value among all the text data as the final candidate data; Determine a semantic duplication penalty value for each final candidate data according to the semantic similarity between the preliminary candidate data and the final candidate data; Determine the score of each final candidate data according to the data attribute and the semantic duplication penalty value of each final candidate data; Determine the data to be removed from all the final candidate data according to the scores of all the final candidate data; Removing the data to be removed from the database; The step of determining the score of each final candidate data according to the data attribute and the semantic duplication penalty value of each final candidate data comprises: When the semantic repetition penalty value of any final candidate data is not equal to the initial value, obtaining at least one preliminary candidate data whose semantic similarity exceeds a threshold value; Comparing the initial score corresponding to any final candidate data and the at least one preliminary candidate data, wherein the initial score is determined by data attributes of the respective data; When the initial score corresponding to any final candidate data is not less than the initial score corresponding to the at least one preliminary candidate data, the semantic repetition penalty value of any final candidate data is reset to the initial value, and the score of any final candidate data is determined based on the data attributes and the semantic repetition penalty value of any final candidate data.

2. The data updating method according to claim 1, characterized in that: Determining the semantic duplication penalty value of each final candidate data according to the semantic similarity between the preliminary candidate data and the final candidate data comprises: For any final candidate data, the number of repetitions between the final candidate data and the preliminary candidate data whose semantic similarity exceeds a threshold in the preliminary candidate data is determined, and a semantic repetition penalty value of the final candidate data is determined according to the number of repetitions.

3. The data updating method according to claim 1, characterized in that: The data attributes include user evaluation, existence time and usage times of each preliminary candidate data, and the method further includes: For each preliminary candidate data, determine the like rate or dislike rate of the preliminary candidate data according to the user evaluation of the preliminary candidate data; For each preliminary candidate data, determine the time decay rate of the preliminary candidate data according to the existence time and decay coefficient of the preliminary candidate data; Arrange all preliminary candidate data in descending order based on their like rate, or in ascending order based on their dislike rate, to obtain the first ranking of each preliminary candidate data; Arrange all preliminary candidate data in ascending order based on their time decay rates to obtain a second ranking for each preliminary candidate data; Arrange all preliminary candidate data in descending order based on the number of times they are used, so as to obtain a third ranking for each preliminary candidate data; An initial score for each preliminary candidate data is determined according to the first ranking, the second ranking, and the third ranking of each preliminary candidate data.

4. The data updating method according to claim 3, characterized in that: Determining the initial score of each preliminary candidate data according to the first ranking, the second ranking and the third ranking of each preliminary candidate data comprises: Obtaining a first weight coefficient representing the importance of user evaluation in the data attribute, a second weight coefficient representing the importance of existence duration in the data attribute, and a third weight coefficient representing the importance of the number of times used in the data attribute; For each preliminary candidate data, respectively determine a first ratio between a first weight coefficient of the preliminary candidate data and a sum of a first ranking and a sorting constant parameter, a second ratio between a second weight coefficient and a sum of a second ranking and the sorting constant parameter, and a third ratio between a third weight coefficient and a sum of a third ranking and the sorting constant parameter; The sum of the first ratio, the second ratio and the third ratio of each preliminary candidate data is determined as an initial score of each preliminary candidate data.

5. The data updating method according to claim 1, characterized in that: Screening out preliminary candidate data that meets preset conditions from the plurality of text data includes: Acquire data attributes of the plurality of text data, wherein the data attributes include user evaluation, existence time, and usage times of each text data; For any one of the plurality of text data, when the downvote rate of the text data in the user evaluation is greater than a preset downvote rate, determining the text data as preliminary candidate data; For any one of the plurality of text data, if the existence time of the text data is greater than a preset time, determining the text data as preliminary candidate data; For any one of the plurality of text data, if the number of times the text data is used is less than a preset number of times, the text data is determined to be preliminary candidate data.

6. The data updating method according to claim 1, characterized in that: Determining the data to be removed from all the final candidate data according to the scores of all the final candidate data includes: Arrange all final candidate data in ascending order according to all scores; A preset number of final candidate data are selected from all the arranged final candidate data as the data to be removed.

7. The data updating method according to claim 1, characterized in that: The method further comprises: After determining the data to be removed from all the final candidate data according to the scores of all the final candidate data, the data to be removed is sent to the user, and the data to be removed approved by the user is stored again in the database; The data to be removed that has not been approved by the user is backed up, and the data to be removed that has not been approved by the user is removed from the database.

8. A data updating device, characterized in that: include: a memory configured to store instructions; A processor is configured to call the instructions from the memory and implement the data updating method according to any one of claims 1 to 7 when executing the instructions.

9. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores instructions, which are used to enable a machine to execute the data updating method according to any one of claims 1 to 7.

Images