Method and device for updating corpus

Abstract

The present invention discloses a method and device for updating a corpus, and relates to the field of computer computing technology. A specific implementation of the method includes: filtering and compressing historical logs to generate new corpora corresponding to the historical logs; using the buried data corresponding to the online corpus and the word score table corresponding to the online corpus to determine the eliminated corpora in the online corpus; adding the new corpus to the online corpus, and deleting the eliminated corpus from the online corpus to update the online corpus. This implementation can ensure that the number of corpora in the online corpus is maintained within a certain range, reduce operating costs, reduce workload, and can also ensure that the corpora in the online corpus are universal, and can continuously follow the hot spots of online customer service system consultation.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for updating a corpus. Background Art

[0002] Input association is a user-side auxiliary tool. Its working mode is: build a corpus. When the user enters information in the front-end interface, it can intercept the user input, predict the user's needs, and select the corpus that best meets the user's needs for recommendation. Therefore, the construction of the corpus is the key to the accuracy of input association.

[0003] In the prior art, there are two ways to construct a corpus: 1. Generative, that is, determining in advance the question sentences that users may ask, then stripping out the entities and adjectives in the question sentences, and then screening out the entities and adjectives that meet the characteristics of the items and filling them into the question sentences; 2. Constructive, that is, screening out the common consultation sentences of users according to certain standards, and then constructing a corpus.

[0004] In the process of realizing the present invention, the inventors found that there are at least the following problems in the prior art: 1. For the generative type, it is necessary to select question sentences from scratch, which will lead to a large workload and extremely high operating costs; 2. For the constructed type, since the consulting questions corresponding to different categories of items are different, the obtained corpus is not universal; 3. For both the generative type and the constructed type, there is no clear elimination mechanism, which causes a surge in the corpus, and there will be outdated corpus that cannot meet user needs. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a method and device for updating a corpus, which can ensure that the amount of corpus in the online corpus is maintained within a certain range, reduce operating costs, reduce workload, and ensure that the corpus in the online corpus is universal and can continuously follow the hot spots of consultation in the online customer service system.

[0006] To achieve the above objective, according to a first aspect of an embodiment of the present invention, a method for updating a corpus is provided.

[0007] A method for updating a corpus according to an embodiment of the present invention includes: filtering and compressing historical logs to generate new corpus corresponding to the historical logs; using the embedded data corresponding to the online corpus and the word score table corresponding to the online corpus to determine the eliminated corpus in the online corpus; adding the new corpus to the online corpus and deleting the eliminated corpus from the online corpus to update the online corpus.

[0008] Optionally, the filtering and compression processing of historical logs to generate new corpus corresponding to the historical logs includes: obtaining historical logs within a first preset time, decrypting the historical logs to obtain plaintext logs; filtering the plaintext logs using preset log filtering rules; compressing the filtered plaintext logs to obtain alternative corpus corresponding to the plaintext logs; and performing model recognition processing on the alternative corpus based on a preset screening model to generate the new corpus.

[0009] Optionally, the preset log filtering rules are used to filter the plaintext logs, including: performing basic filtering on the plaintext logs to filter out system default messages, user-related information, URLs, pictures and special characters in the plaintext logs, and replacing emoticons in the plaintext logs; performing intra-session deduplication processing on the plaintext logs to filter out duplicate sessions in each plaintext log; performing length filtering on the plaintext logs to filter out logs in the plaintext logs that do not meet the preset character length; performing number filtering on the plaintext logs to filter out logs in the plaintext logs that appear less than a preset number of times within a second preset time.

[0010] Optionally, compressing the plaintext log after filtering to obtain the candidate corpus corresponding to the plaintext log includes: compressing the plaintext log after filtering according to the number of times the plaintext log after filtering appears within a second preset time to obtain the candidate corpus.

[0011] Optionally, based on a preset screening model, the candidate corpus is subjected to model recognition processing to generate the newly added corpus, including: inputting the candidate corpus into a negative emotion recognition model to identify negative emotion corpus, inputting the candidate corpus into a sensitive compliance model to identify non-compliant corpus; deleting the negative emotion corpus and the non-compliant corpus from the candidate corpus to generate the newly added corpus.

[0012] Optionally, before using the embedded data corresponding to the online corpus and the word score table corresponding to the online corpus to determine the eliminated corpus in the online corpus, the method also includes: acquiring at least one online corpus in the online corpus, segmenting the at least one online corpus, and obtaining at least one online corpus segmentation; for the online corpus segmentations in the at least one online corpus segmentation, counting the number of times the online corpus segmentation appears in the online corpus, and determining the word score of the online corpus segmentation based on the counted number of times, so as to generate a word score table corresponding to the online corpus.

[0013] Optionally, the buried data includes: exposure data and click data; and the use of the buried data and the word score table corresponding to the online corpus to determine the eliminated corpus in the online corpus includes: for the online corpus in the at least one online corpus, obtaining the number of exposures corresponding to the online corpus from the exposure data, and obtaining the number of clicks corresponding to the online corpus from the click data; judging whether the online prediction meets the preset elimination conditions according to the number of exposures corresponding to the online corpus, the number of clicks corresponding to the online corpus and the word score table, and if so, determining that the online prediction is an eliminated prediction to obtain the eliminated corpus in the online corpus.

[0014] Optionally, judging whether the online prediction meets the preset elimination condition according to the number of exposures corresponding to the online corpus, the number of clicks corresponding to the online corpus and the word score table, and if so, determining that the online prediction is an eliminated prediction, includes: if the number of exposures corresponding to the online corpus is zero, determining that the online corpus is eliminated corpus; if the number of exposures corresponding to the online corpus is greater than a first preset number of exposures, and the number of clicks corresponding to the online corpus is less than a preset number of clicks, determining that the online corpus is eliminated corpus; if the number of exposures corresponding to the online corpus is less than a second preset number of exposures and greater than zero, calculating the corpus score corresponding to the online corpus using the word score table, and determining that the online corpus is eliminated corpus when the corpus score is less than the preset score.

[0015] Optionally, after generating the new corpus corresponding to the historical log, the method also includes: obtaining features corresponding to the new corpus, establishing a corresponding relationship between the acquired features and the new corpus, the features including category type, brand type, and / or consultation entry type; and, tracking the new corpus to monitor the exposure times and click times corresponding to the new corpus.

[0016] To achieve the above objective, according to a second aspect of an embodiment of the present invention, a device for updating a corpus is provided.

[0017] A device for updating a corpus according to an embodiment of the present invention includes: a generation module, which is used to filter and compress historical logs to generate new corpus corresponding to the historical logs; a determination module, which is used to use the embedded data corresponding to the online corpus and the word score table corresponding to the online corpus to determine the eliminated corpus in the online corpus; and an update module, which is used to add the new corpus to the online corpus and delete the eliminated corpus from the online corpus to update the online corpus.

[0018] Optionally, the generation module is also used to: obtain historical logs within a first preset time, decrypt the historical logs to obtain plaintext logs; filter the plaintext logs using preset log filtering rules; compress the filtered plaintext logs to obtain alternative corpora corresponding to the plaintext logs; perform model recognition on the alternative corpora based on a preset screening model to generate the newly added corpora.

[0019] Optionally, the generation module is also used to: perform basic filtering on the plaintext log, filter out system default messages, user-related information, URLs, pictures and special characters in the plaintext log, and replace emoticons in the plaintext log; perform intra-session deduplication processing on the plaintext log, filter out duplicate sessions in each plaintext log; perform length filtering on the plaintext log, filter out logs in the plaintext log that do not meet a preset character length; perform number filtering on the plaintext log, filter out logs in the plaintext log that appear less than a preset number of times within a second preset time.

[0020] Optionally, the generating module is further used to: compress the plaintext log after the filtering process according to the number of times the plaintext log after the filtering process appears within a second preset time to obtain the candidate corpus.

[0021] Optionally, the generation module is also used to: input the candidate corpus into a negative emotion recognition model to identify negative emotion corpus, input the candidate corpus into a sensitive compliance model to identify non-compliant corpus; delete the negative emotion corpus and the non-compliant corpus from the candidate corpus to generate the newly added corpus.

[0022] Optionally, the determination module is also used to: obtain at least one online corpus in the online corpus, segment the at least one online corpus, and obtain at least one online corpus segmentation; for the online corpus segmentation in the at least one online corpus segmentation, count the number of times the online corpus segmentation appears in the online corpus, and determine the word score of the online corpus segmentation based on the counted number of times, so as to generate a word score table corresponding to the online corpus.

[0023] Optionally, the buried data includes: exposure data and click data; and the determination module is also used to: for an online corpus in the at least one online corpus, obtain the number of exposures corresponding to the online corpus from the exposure data, and obtain the number of clicks corresponding to the online corpus from the click data; based on the number of exposures corresponding to the online corpus, the number of clicks corresponding to the online corpus and the word score table, determine whether the online prediction meets the preset elimination conditions, and if so, determine that the online prediction is an eliminated prediction, so as to obtain the eliminated corpus in the online corpus.

[0024] Optionally, the determination module is also used to: if the number of exposures corresponding to the online corpus is zero, determine that the online corpus is an eliminated corpus; if the number of exposures corresponding to the online corpus is greater than a first preset number of exposures, and the number of clicks corresponding to the online corpus is less than a preset number of clicks, determine that the online corpus is an eliminated corpus; if the number of exposures corresponding to the online corpus is less than a second preset number of exposures and greater than zero, calculate the corpus score corresponding to the online corpus using the word score table, and if the corpus score is less than the preset score, determine that the online corpus is an eliminated corpus.

[0025] Optionally, the generation module is also used to: obtain features corresponding to the newly added corpus, establish a corresponding relationship between the acquired features and the newly added corpus, the features including category type, brand type, and / or consultation entry type; and, perform tracking on the newly added corpus to monitor the exposure times and click times corresponding to the newly added corpus.

[0026] To achieve the above objective, according to a third aspect of an embodiment of the present invention, an electronic device is provided.

[0027] An electronic device according to an embodiment of the present invention includes: one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method for updating the corpus according to the embodiment of the present invention.

[0028] To achieve the above objective, according to a fourth aspect of an embodiment of the present invention, a computer-readable medium is provided.

[0029] A computer-readable medium according to an embodiment of the present invention stores a computer program, and when the program is executed by a processor, the method for updating a corpus according to an embodiment of the present invention is implemented.

[0030] An embodiment of the above invention has the following advantages or beneficial effects: generating new corpus corresponding to historical logs, being able to screen and repair consultation texts that users have recently paid close attention to, obtaining new corpus, and adding it to the online corpus, so that the corpus in the online corpus is universal and can continuously follow the hot spots of consultation in the online customer service system; using the buried data and word score table corresponding to the online corpus to determine the eliminated corpus, automatically selecting the corpus that needs to be eliminated in the online corpus, so that outdated or low-quality corpus can be deleted; adding new corpus to the online corpus and deleting the eliminated corpus from the online corpus can ensure that the number of corpora in the online corpus is maintained within a certain range, reduce operating costs, reduce workload, and avoid the problem of a surge in corpus and the existence of expired corpus that cannot meet user needs due to the lack of a clear elimination mechanism in the prior art.

[0031] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0033] Figure 1 is a schematic diagram of main steps of a method for updating a corpus according to an embodiment of the present invention;

[0034] Figure 2 is a schematic diagram of the main process of a method for generating new corpus corresponding to a historical log according to an embodiment of the present invention;

[0035] Figure 3 is a schematic diagram of the main process of a method for determining eliminated corpora in an online corpus according to an embodiment of the present invention;

[0036] Figure 4 is a schematic structural diagram of a system for updating a corpus according to an embodiment of the present invention;

[0037] Figure 5 is a schematic diagram of main modules of a device for updating a corpus according to an embodiment of the present invention;

[0038] Figure 6 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0039] Figure 7 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0041] The key parts of the corpus are the generation of corpus and the timely elimination of corpus. Currently, the corpus can be constructed using generative and constructive methods. Among them, the generative method is to determine the question sentences that users may ask in advance, then strip the entities and adjectives in the sentences, and then select the entities and adjectives that meet the characteristics of the items to fill in these sentences. This generative construction method can generate a large amount of corpus in batches, with strong personalization, but the operating cost is extremely high, which is specifically reflected in: the screening of question sentences will vary greatly with the different categories of items, and the richness and diversity of the corpus will lead to a lot of work in selecting from scratch; the need to maintain variable parameters such as entities and adjectives in the corpus by oneself has created obstacles to the difficulty of operation; the elimination of corpus is more difficult, and there is no clear elimination mechanism, which will lead to an explosive growth of corpus. The constructive method is to select common user consultation sentences based on certain standards, and then construct a corpus. For the constructive method, different categories of items have different consultation questions, resulting in the corpus not being universal; in addition, the lack of dynamic update and elimination mechanism will cause the corpus to expire and fail to meet user needs.

[0042] In order to solve the above problems, the embodiments of the present invention provide a method and device for updating a corpus, which can continuously filter out high-quality new corpora from massive texts and automatically eliminate low-quality old corpora according to the actual operation of the corpora. Figure 1 FIG. 1 is a schematic diagram of the main steps of a method for updating a corpus according to an embodiment of the present invention. Figure 1 As shown, the main steps of the method for updating the corpus may include steps S101 to S103.

[0043] Step S101: filtering and compressing the historical logs to generate new corpus corresponding to the historical logs.

[0044] In an embodiment of the present invention, recent historical logs (which may be, but are not limited to, logs from the last 7 days) are directly processed to obtain newly added corpus. The advantage of doing so is that the consulting texts that users have recently paid close attention to can be screened and repaired to obtain newly added corpus, which is added to the online corpus. Historical logs can be obtained from the log library, which can continuously add new corpus to the online corpus, so that the corpus in the online corpus constantly follows the hot spots of online customer service system consultation. Of course, before the historical logs become corpus, it is necessary to filter and compress the historical logs, so that the inferior texts in the historical logs can be deleted, and the logs can be compressed into corpus form to become formal corpus.

[0045] In addition, the log library in the embodiment of the present invention can be a conversation log library. For example, when a user is interested in an item or an event, he can consult the item or the event on the platform, so that the conversation material between the user and the customer service can be obtained, and then the conversation log library can be obtained. Then, the historical log in the embodiment of the present invention can be a historical conversation log, and the plaintext log can be a plaintext conversation log. Of course, the historical log is not limited to the historical conversation log, and the plaintext log is not limited to the plaintext conversation log. In addition, the log library can also be a chat log library, such as the chat records between users, and the chat log library obtained. It should be noted that the chat records between users involve user personal information, so the chat records in the chat log library are records that have been agreed to by the user.

[0046] Step S102: using the embedded data corresponding to the online corpus and the word score table corresponding to the online corpus, determine the eliminated corpus in the online corpus.

[0047] The buried data may include exposure data and click data, which are obtained by monitoring the online corpus in the online corpus. Among them, exposure data refers to the number of exposures of online corpus obtained by monitoring, and the specific form is: at a certain time, a certain user inputs a certain word, and a certain corpus is exposed; exposure refers to the corpus that appears above the input box after the user enters the text, and the upper limit of the number of corpora that appear each time can be set to 3. Click data refers to the number of clicks on online corpus obtained by monitoring, and the specific form is: at a certain time, a certain user enters a certain word and clicks on a certain exposed corpus; click means that the user clicks on a certain corpus exposed on the input box. Therefore, from the exposure data and click data corresponding to the online corpus, the number of exposures corresponding to the online corpus, that is, the number of times the online corpus is exposed, can be obtained, and the number of clicks corresponding to the online corpus, that is, the number of times the online corpus is clicked, can also be obtained.

[0048] The word score table is obtained by scoring the words in the online corpus. Using the word score table, the corpus score of each online corpus can be calculated, and then it can be determined whether the online corpus needs to be eliminated based on the corpus score. In a reference embodiment of the present invention, the method for generating a word score table may include: obtaining at least one online corpus in the online corpus, segmenting the at least one online corpus, and obtaining at least one online corpus segmentation; for the online corpus segmentation in the at least one online corpus segmentation, counting the number of times the online corpus segmentation appears in the online corpus, and determining the word score of the online corpus segmentation based on the counted number of times, so as to generate a word score table corresponding to the online corpus.

[0049] Specifically, first obtain all the online corpora in the online corpus; then segment each online corpus to obtain all the online corpus segmentations; then for each online corpus segmentation, count the number of times the online corpus segmentation appears in the online corpus, and score the online corpus segmentation according to the counted number of times to obtain the word score of the online corpus segmentation, so that the word score of each online corpus segmentation can be obtained, and then generate the word score table corresponding to the online corpus. It should be noted that the more times an online corpus segmentation appears in the online corpus, the higher the word score of the online corpus segmentation. In addition, after obtaining all the online corpus segmentations, it is necessary to filter out meaningless words that appear more frequently, such as, I, of, and so on. In the embodiment of the present invention, a meaningless word list can be generated in advance, so that all online corpus segmentations can be filtered using the meaningless word list.

[0050] The exposure times and click times of the online corpus can be obtained by using the embedding data corresponding to the online corpus. The corpus scores corresponding to the online corpus can be obtained by using the word score table corresponding to the online corpus. Therefore, the online corpus can be scored based on the exposure times, click times and corpus scores of the online corpus, and the corpus that needs to be eliminated in the online corpus can be automatically selected.

[0051] Step S103: Add the newly added corpus to the online corpus and delete the eliminated corpus from the online corpus to update the online corpus.

[0052] Through step S101, the newly added corpus corresponding to the historical log can be obtained, and through step S102, the eliminated corpus in the online corpus can be obtained. Then, in step S103, the newly added corpus is added to the online corpus, and the eliminated corpus is deleted from the online corpus, thereby completing the update of the online corpus.

[0053] The technical solution for updating the corpus of the embodiment of the present invention generates new corpus corresponding to the historical log, can screen and repair the consultation texts that users have paid close attention to recently, obtain new corpus, and add it to the online corpus, so that the corpus in the online corpus has universality and can continuously follow the hot spots of consultation in the online customer service system; use the buried data and word score table corresponding to the online corpus to determine the eliminated corpus, and automatically select the corpus that needs to be eliminated in the online corpus, so that outdated or low-quality corpus can be deleted; add the new corpus to the online corpus and delete the eliminated corpus from the online corpus, which can ensure that the number of corpora in the online corpus is maintained within a certain range, reduce operating costs, reduce workload, and avoid the problem of a surge in corpus and the existence of expired corpus that cannot meet user needs due to the lack of a clear elimination mechanism in the prior art.

[0054] Updating the corpus can include the following two parts: supplementing with new corpus and deleting obsolete corpus. For the supplement of new corpus, it mainly depends on the text processing ability of the big data system, and the core is to delete the inferior text in the historical log. In the reference embodiment of the present invention, filtering and compressing the historical logs to generate new corpus corresponding to the historical logs can include: obtaining the historical logs within the first preset time, decrypting the historical logs, and obtaining the plaintext logs; using the preset log filtering rules, filtering the plaintext logs; compressing the filtered plaintext logs to obtain the candidate corpus corresponding to the plaintext logs; based on the preset screening model, model recognition processing is performed on the candidate corpus to generate new corpus.

[0055] In the method for generating new corpus, firstly, historical logs within the first preset time are obtained from the log library, such as the historical logs of the most recent week. Considering that the logs stored in the log library are encrypted in order to protect user privacy, after obtaining the historical logs, the obtained historical logs need to be decrypted to obtain plaintext logs. Since the amount of data of the plaintext logs obtained after decryption is huge, the plaintext logs can be filtered using preset log filtering rules to quickly filter out low-quality texts in the plaintext logs. At this time, there are still a large number of repeated texts. The plaintext logs after filtering can be compressed, and the repeated texts can be merged into one record to further reduce the number of logs and obtain the alternative corpus corresponding to the plaintext logs. Finally, the alternative corpus can be model-recognized based on the preset screening model to identify the non-compliant corpus in the alternative corpus and achieve the purpose of filtering again. Because the preset screening model is inefficient and extremely time-consuming, in an embodiment of the present invention, the preset log filtering rules are first used for filtering processing, so that a large amount of low-quality text can be quickly screened out; then the repeated text in the plaintext log after filtering is merged into one record to obtain alternative corpus, so that the amount of data can be further reduced; finally, the preset screening model is used to perform model recognition processing on the alternative corpus, and the non-compliant corpus in the alternative corpus is identified, and finally the newly added corpus is obtained.

[0056] The preset log filtering rules may include: basic filtering, deduplication processing, length filtering, and number filtering. In the reference embodiment of the present invention, the preset log filtering rules are used to filter the plain text logs, which may include:

[0057] (1) Perform basic filtering on plain text logs, filter out system default messages, user-related information, URLs, pictures, and special characters in plain text logs, and replace emoticons in plain text logs. Among them, the system default message can be the system default reply message that has a weak correlation with user input, so the system default message needs to be filtered out; user-related information includes: address, phone number, user name, credit card, social account and other information related to user privacy, so user-related information needs to be filtered out; URLs, pictures, and special characters have a weak correlation with user input associations, so URLs, pictures, and special characters need to be filtered out; emoticons can express the user's tone and attitude, which is of great significance, so emoticons need to be replaced with a certain format;

[0058] (2) Perform intra-session deduplication on plaintext logs and filter out duplicate sessions in each plaintext log. In a log, a user may copy and paste the text they have sent and send it again, thereby emphasizing the importance of the text sent by the user, but it does not mean that the text has the same effect in other logs. Therefore, the text needs to be deduplicated and only retained once;

[0059] (3) Perform length filtering on plain text logs and filter out those that do not meet the preset character length. Considering that shorter logs do not need to be input, and longer logs have too many personalized details and are difficult to meet user needs, a preset character length can be set based on comprehensive considerations. For example, the preset character length can be set within a range of 5 to 20 characters, but is not limited to the preset character length. The length of each plain text log is analyzed and logs that are no longer within the preset character length range are filtered out;

[0060] (4) Filter the plaintext logs by the number of times, and filter out the logs in the plaintext logs that appear less than the preset number of times within the second preset time. For the plaintext logs within the first preset time, for each plaintext log, check the number of times the plaintext log appears within the second preset time. If the number of times is greater than or equal to the preset number, it can be considered that the plaintext log has generalization ability and can be used in multiple scenarios. If the number of times is less than the preset number, the plaintext log can be eliminated. The first preset time is less than the second preset time. For example, the first preset time can be the last 7 days, and the second preset time can be the last 20 days.

[0061] The above (1) to (4) describe the use of preset log filtering rules to filter plaintext logs. It should be noted that the execution order of (1) to (4) can be adjusted according to actual conditions, and the embodiments of the present invention do not limit this. After filtering the plaintext log, a large amount of low-quality text in the log can be filtered out, but considering that there are still a large number of repeated texts, it is necessary to compress the plaintext log after filtering. Therefore, in the reference embodiment of the present invention, compressing the plaintext log after filtering to obtain the candidate corpus corresponding to the plaintext log can include: compressing the plaintext log after filtering according to the number of times the plaintext log after filtering appears within the second preset time to obtain the candidate corpus. Counting the number of times the plaintext log after filtering appears within the second preset time, and then merging the repeated plaintext logs into one record according to the number of times they appear, to obtain the candidate corpus corresponding to the plaintext log, thereby further reducing the number of logs.

[0062] After filtering the plain text log, a large amount of low-quality text in the log can be filtered out, but there are still negative emotion logs and non-compliant logs in the plain text log obtained after filtering, so further screening is required. Therefore, in the reference embodiment of the present invention, based on the preset screening model, the candidate corpus is subjected to model recognition processing to generate new corpus, which may include: inputting the candidate corpus into the negative emotion recognition model to identify the negative emotion corpus, inputting the candidate corpus into the sensitive compliance model to identify the non-compliant corpus; deleting the negative emotion corpus and the non-compliant corpus from the candidate corpus to generate new corpus.

[0063] Both the negative emotion recognition model and the sensitive compliance model are deep learning models and are mature technologies. The specific construction process of the models will not be described here. The negative emotion recognition model is used to identify corpus with negative emotions, anger, and dissatisfaction, which are named negative emotion corpus; the sensitive compliance model is used to identify words that do not comply with regulations or are advertising promotion or harassment. If the corpus contains these words, the corpus is named non-compliant corpus. After identifying the negative emotion corpus and non-compliant corpus, these corpus are deleted from the candidate corpus, and the new corpus is obtained.

[0064] Figure 2 FIG. 1 is a schematic diagram of the main process of a method for generating new corpus corresponding to a historical log according to an embodiment of the present invention. Figure 2 As shown, the main process of the method for generating new corpus corresponding to the historical log may include:

[0065] Step S201, obtaining historical logs within a first preset time, decrypting the historical logs, and obtaining plaintext logs;

[0066] Step S202, performing basic filtering on the plain text log, filtering out the system default messages, user related information, URLs, pictures and special characters in the plain text log, and replacing the emoticons in the plain text log;

[0067] Step S203, performing session deduplication processing on the plain text log, filtering out duplicate sessions in each plain text log;

[0068] Step S204, performing length filtering on the plain text logs, filtering out the plain text logs that do not meet the preset character length;

[0069] Step S205, filtering the plain text logs by number of times, filtering out the plain text logs whose number of times in the second preset time is less than the preset number of times;

[0070] Step S206, compressing the plaintext log after filtering according to the number of times the plaintext log after filtering appears within the second preset time to obtain a candidate corpus;

[0071] Step S207, inputting the candidate corpus into the negative emotion recognition model to identify the negative emotion corpus, and inputting the candidate corpus into the sensitive compliance model to identify the non-compliant corpus;

[0072] Step S208: delete the negative sentiment corpus and the non-compliant corpus from the candidate corpus to generate new corpus.

[0073] Among them, steps S202 to S205 are used to filter the plain text logs. The specific execution order can be adjusted according to the actual situation. The embodiment of the present invention does not limit this. In the specific filtering method, it can be processed from multiple angles such as basic filtering, deduplication processing, length filtering, and frequency filtering, and low-quality text can be deleted to improve the accuracy of the newly added corpus. In addition, in the method of generating the newly added corpus, the negative emotion recognition model and the sensitive compliance model are also used to delete the identified negative emotion corpus and non-compliant corpus, further improving the accuracy of the newly added corpus.

[0074] By using the buried data and word score table corresponding to the online corpus, the eliminated corpus is determined, and the corpus that needs to be eliminated in the online corpus is automatically selected, so that the outdated or low-quality corpus can be deleted. In the reference embodiment of the present invention, the buried data may include exposure data and click data; and, using the buried data and word score table corresponding to the online corpus, the eliminated corpus in the online corpus can include: for the online corpus in at least one online corpus, the number of exposures corresponding to the online corpus is obtained from the exposure data, and the number of clicks corresponding to the online corpus is obtained from the click data; according to the number of exposures corresponding to the online corpus, the number of clicks corresponding to the online corpus and the word score table, it is judged whether the online expectation meets the preset elimination conditions, and if so, the online expectation is determined to be an eliminated expectation, so as to obtain the eliminated corpus in the online corpus.

[0075] Among them, the preset elimination conditions can be: (1) if the number of exposures corresponding to the online corpus is zero, the online corpus is determined to be eliminated corpus; (2) if the number of exposures corresponding to the online corpus is greater than a first preset number of exposures, and the number of clicks corresponding to the online corpus is less than a preset number of clicks, the online corpus is determined to be eliminated corpus; (3) if the number of exposures corresponding to the online corpus is less than a second preset number of exposures and greater than zero, the corpus score corresponding to the online corpus is calculated using the word score table. If the corpus score is less than the preset score, the online corpus is determined to be eliminated corpus.

[0076] Analyze each online corpus in the online corpus, obtain the exposure times corresponding to the online corpus from the exposure data, and obtain the click times corresponding to the online corpus from the click data. If the exposure times corresponding to a certain online corpus is zero, it means that the online corpus has not been exposed, and the online corpus can be directly determined to be an eliminated corpus. If the exposure times corresponding to a certain online corpus are greater than the first preset exposure times, and the click times corresponding to the online corpus are less than the preset click times, it means that the online corpus has been exposed a high number of times but clicked a low number of times, and it also means that the online corpus will frequently appear in the user interface but does not meet user needs, resulting in a decrease in click rate, so the online corpus can be determined to be an eliminated corpus. If the exposure times corresponding to a certain online corpus are less than the second preset exposure times and greater than zero, the corpus score corresponding to the online corpus can be calculated using the word score table, and if the corpus score is less than the preset score, the online corpus is determined to be an eliminated corpus. For online corpus whose exposure times are less than the second preset exposure times and greater than zero, it is necessary to use the word score table to calculate the corpus score corresponding to the online corpus, and then determine whether the corpus score is less than the preset score. If so, the online corpus is considered to be eliminated. The word score table is used to calculate the corpus score corresponding to the online corpus, specifically: segment the online corpus, use the word score table to find the word score corresponding to each segmentation, and average the word scores found, and the average score obtained is the corpus score corresponding to the online corpus.

[0077] Figure 3 FIG. 1 is a schematic diagram of the main process of a method for determining eliminated corpora in an online corpus according to an embodiment of the present invention. Figure 3 As shown, the main process of the method for determining the eliminated corpus in the online corpus may include:

[0078] Step S301, obtaining exposure data and click data corresponding to the online corpus;

[0079] Step S302, obtaining at least one online corpus in the online corpus, performing word segmentation on the at least one online corpus, and obtaining at least one online corpus word segmentation;

[0080] Step S303, for the online corpus segmentation in the at least one online corpus segmentation, counting the number of times the online corpus segmentation appears in the online corpus, and determining the word score of the online corpus segmentation according to the counted number of times, so as to generate a word score table corresponding to the online corpus;

[0081] Step S304, randomly selecting an online corpus from at least one online corpus;

[0082] Step S305, obtaining the exposure times corresponding to the online corpus from the exposure data, and obtaining the click times corresponding to the online corpus from the click data;

[0083] Step S306: if the exposure times corresponding to the online corpus is zero, the online corpus is determined to be eliminated corpus;

[0084] Step S307: if the exposure number corresponding to the online corpus is greater than the first preset exposure number, and the click number corresponding to the online corpus is less than the preset click number, then the online corpus is determined to be eliminated corpus;

[0085] Step S308: if the exposure number corresponding to the online corpus is less than the second preset exposure number and greater than zero, the corpus score corresponding to the online corpus is calculated using the word score table, and if the corpus score is less than the preset score, the online corpus is determined to be eliminated corpus;

[0086] Step S309, determining whether all online corpora in the online corpus have been analyzed, if so, the process ends, if not, executing step S304.

[0087] It should be noted that step S302 and step S303 are used to generate a word score table corresponding to the online corpus. Their execution order can be adjusted according to the actual situation, and they can be executed before step S308. In addition, steps S306 to S308 are used to determine the eliminated corpus in the online corpus. Steps S306 to S308 can be executed simultaneously, or in the order from step S306 to step S308. The specific execution order can be adjusted according to the actual situation, and there is no limitation on this. In the specific method of determining the eliminated corpus, the exposure and click times of the online corpus can be obtained by using the buried point data corresponding to the online corpus, and the corpus score corresponding to the online corpus can be obtained by using the word score table corresponding to the online corpus. Therefore, the online corpus is scored in combination with the exposure times, click times and corpus scores of the online corpus, and the corpus that needs to be eliminated in the online corpus is automatically selected, which solves the problem of the surge of corpus and the existence of expired corpus that cannot meet user needs in the prior art due to the lack of a clear elimination mechanism.

[0088] After the new corpus is generated, features related to the application scenario can be added to the new corpus to provide an association basis in the corpus search stage, so that the corpus is more matched with the consulting scenario when it is associated, and the obtained associated corpus is more accurate. Therefore, in the reference embodiment of the present invention, after the new corpus corresponding to the historical log is generated, the method for updating the corpus can also include: obtaining features corresponding to the new corpus, establishing a corresponding relationship between the obtained features and the new corpus, the features include category type, brand type, and / or consulting entry type; and, burying points for the new corpus, monitoring the number of exposures and clicks corresponding to the new corpus.

[0089] Among them, the features may include: category type, brand type, and / or consultation entry type. Category type refers to the category corresponding to the item in the newly added corpus. The category information library can classify the corpus according to the corpus when the user consults different items, and match the category type for the corpus; brand type refers to the brand corresponding to the item in the newly added corpus. The item information library stores the correspondence between the corpus and different items. For different items, the corpus is classified to obtain the brand type corresponding to the corpus; consultation entry type refers to the consultation entry corresponding to the newly added corpus. For example, if the user enters from the item details page, each text will be marked with the item details entry before the session ends. If the user enters from after-sales consultation, each text will be marked with the after-sales consultation entry. In addition, after the newly added corpus is generated, the newly added corpus needs to be added to the online corpus, that is, the newly added corpus can be regarded as online corpus, so its corresponding exposure data and click data can be obtained as the exposure click feature of the newly added corpus. In the embodiment of the present invention, after adding corresponding features to the newly added corpus, when the user inputs association, the corresponding corpus will be retrieved from the online database according to the added corpus category type, corpus brand type, corpus consultation entry type, and corpus exposure click features.

[0090] Figure 4 4 is a schematic diagram of the structure of a system for updating a corpus according to an embodiment of the present invention. Figure 4 In the system, the log library is mainly used to continuously add new corpora to the online corpus, so that the corpora in the online corpus can keep up with the hot topics of online customer service system consultation. Before the log becomes a new corpus, it will be filtered, deduplicated, intercepted, replaced, compressed, desensitized, and violated. Detection and a series of operations will be performed on the log, and then it will become a new corpus. It is also necessary to eliminate the online corpus in the online corpus. Specifically, the online corpus will be scored according to the number of corpus exposures, the number of corpus clicks, and the corpus score indicators to obtain the eliminated corpus. The online corpus is updated with the new corpus and the eliminated corpus. Specifically, the new corpus is added to the online corpus, and the eliminated corpus is deleted from the online corpus. This ensures that the total amount of the online corpus meets a certain range of quantities. In addition, it is also necessary to bury data points in the online corpus, count the number of exposures and clicks of the online corpus, and obtain the exposure data and click data corresponding to the online corpus.

[0091] According to the technical solution for updating the corpus of the embodiment of the present invention, new corpus corresponding to the historical log is generated, and the consultation texts that users have paid close attention to recently can be screened and repaired to obtain new corpus, which is added to the online corpus, so that the corpus in the online corpus has universality and can continuously follow the hot spots of consultation in the online customer service system; the eliminated corpus is determined by using the buried data and word score table corresponding to the online corpus, and the corpus that needs to be eliminated in the online corpus is automatically selected, so that outdated or low-quality corpus can be deleted; the new corpus is added to the online corpus, and the eliminated corpus is deleted from the online corpus, which can ensure that the number of corpora in the online corpus is maintained within a certain range, reduce operating costs, reduce workload, and avoid the problem of a surge in corpora and the existence of expired corpora that cannot meet user needs due to the lack of a clear elimination mechanism in the prior art.

[0092] Figure 5 Schematic diagram of main modules of the apparatus for updating the corpus according to an embodiment of the present invention. Figure 5 As shown, the main modules of the apparatus 500 for updating a corpus may include: a generating module 501 , a determining module 502 and an updating module 503 .

[0093] Among them, the generation module 501 can be used to filter and compress the historical logs to generate new corpus corresponding to the historical logs; the determination module 502 can be used to use the embedded data corresponding to the online corpus and the word score table corresponding to the online corpus to determine the eliminated corpus in the online corpus; the update module 503 can be used to add the new corpus to the online corpus and delete the eliminated corpus from the online corpus to update the online corpus.

[0094] In an embodiment of the present invention, the generation module 501 can also be used to: obtain historical logs within a first preset time, decrypt the historical logs, and obtain plaintext logs; use preset log filtering rules to filter the plaintext logs; compress the filtered plaintext logs to obtain alternative corpora corresponding to the plaintext logs; based on a preset screening model, perform model recognition processing on the alternative corpora to generate new corpora.

[0095] In an embodiment of the present invention, the generation module 501 can also be used to: perform basic filtering on the plaintext logs, filter out the system default messages, user-related information, URLs, pictures and special characters in the plaintext logs, and replace the emoticons in the plaintext logs; perform intra-session deduplication processing on the plaintext logs, filter out duplicate sessions in each plaintext log; perform length filtering on the plaintext logs, filter out the logs in the plaintext logs that do not meet the preset character length; perform number filtering on the plaintext logs, filter out the logs in the plaintext logs that appear less than the preset number of times within the second preset time.

[0096] In the embodiment of the present invention, the generating module 501 may also be used to: compress the plain text log after filtering according to the number of times the plain text log after filtering appears within the second preset time to obtain the candidate corpus.

[0097] In an embodiment of the present invention, the generation module 501 can also be used to: input the alternative corpus into a negative emotion recognition model to identify negative emotion corpus, input the alternative corpus into a sensitive compliance model to identify non-compliant corpus; delete the negative emotion corpus and non-compliant corpus from the alternative corpus to generate new corpus.

[0098] In an embodiment of the present invention, the determination module 502 can also be used to: obtain at least one online corpus in the online corpus, segment the at least one online corpus, and obtain at least one online corpus segmentation; for the online corpus segmentation in the at least one online corpus segmentation, count the number of times the online corpus segmentation appears in the online corpus, and determine the word score of the online corpus segmentation based on the counted number of times to generate a word score table corresponding to the online corpus.

[0099] In the embodiment of the present invention, the embedding data may include: exposure data and click data. And, the determination module 502 may also be used to: for the online corpus in at least one online corpus, obtain the exposure times corresponding to the online corpus from the exposure data, and obtain the click times corresponding to the online corpus from the click data; according to the exposure times corresponding to the online corpus, the click times corresponding to the online corpus and the word score table, determine whether the online prediction meets the preset elimination condition, and if so, determine that the online prediction is an eliminated prediction, so as to obtain the eliminated corpus in the online corpus.

[0100] In the embodiment of the present invention, the determination module 502 may also be used for: if the number of exposures corresponding to the online corpus is zero, determining that the online corpus is an eliminated corpus; if the number of exposures corresponding to the online corpus is greater than a first preset number of exposures, and the number of clicks corresponding to the online corpus is less than a preset number of clicks, determining that the online corpus is an eliminated corpus; if the number of exposures corresponding to the online corpus is less than a second preset number of exposures and greater than zero, calculating the corpus score corresponding to the online corpus using the word score table, and if the corpus score is less than the preset score, determining that the online corpus is an eliminated corpus.

[0101] In an embodiment of the present invention, the generation module 501 can also be used to: obtain features corresponding to the newly added corpus, establish a corresponding relationship between the acquired features and the newly added corpus, the features include category type, brand type, and / or consultation entry type; and, track the newly added corpus to monitor the exposure times and click times corresponding to the newly added corpus.

[0102] From the above description, it can be seen that the device for updating the corpus of the embodiment of the present invention generates new corpus corresponding to the historical log, can screen and repair the consultation texts that users have paid close attention to recently, obtain new corpus, and add it to the online corpus, so that the corpus in the online corpus is universal and can continuously follow the hot spots of consultation in the online customer service system; use the buried data and word score table corresponding to the online corpus to determine the eliminated corpus, and automatically select the corpus that needs to be eliminated in the online corpus, so that outdated or low-quality corpus can be deleted; add the new corpus to the online corpus and delete the eliminated corpus from the online corpus, which can ensure that the number of corpora in the online corpus is maintained within a certain range, reduce operating costs, reduce workload, and avoid the problem of corpus explosion and the existence of expired corpus that cannot meet user needs due to the lack of a clear elimination mechanism in the prior art.

[0103] Figure 6 An exemplary system architecture 600 is shown to which the method for updating a corpus or the apparatus for updating a corpus according to an embodiment of the present invention can be applied.

[0104] like Figure 6 As shown, system architecture 600 may include terminal devices 601, 602, 603, network 604 and server 605. Network 604 is used to provide a medium for communication links between terminal devices 601, 602, 603 and server 605. Network 604 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0105] Users can use terminal devices 601, 602, and 603 to interact with server 605 through network 604 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 601, 602, and 603, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only examples).

[0106] The terminal devices 601 , 602 , and 603 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers, etc.

[0107] The server 605 may be a server that provides various services, such as a backend management server (only an example) that provides support for shopping websites browsed by users using the terminal devices 601, 602, and 603. The backend management server may analyze and process the received data such as product information query requests, and feed back the processing results (such as target push information, product information - only an example) to the terminal device.

[0108] It should be noted that the method for updating the corpus provided in the embodiment of the present invention is generally executed by the server 605 , and accordingly, the device for updating the corpus is generally disposed in the server 605 .

[0109] It should be understood that Figure 6 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided as required.

[0110] Reference below Figure 7 , which shows a schematic diagram of the structure of a computer system 700 of a terminal device suitable for implementing an embodiment of the present invention. Figure 7 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0111] like Figure 7 As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the system 700 are also stored. The CPU 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0112] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage section 708 as needed.

[0113] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the central processing unit (CPU) 701, the above-mentioned functions defined in the system of the present invention are executed.

[0114] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0115] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0116] The modules involved in the embodiments of the present invention may be implemented by software or hardware. The modules described may also be set in a processor, for example, it may be described as: a processor includes a generation module, a determination module and an update module. The names of these modules do not constitute a limitation on the modules themselves in some cases. For example, the generation module may also be described as "a module for filtering and compressing historical logs to generate new corpus corresponding to the historical logs".

[0117] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs. When the above one or more programs are executed by a device, the device includes: filtering and compressing the historical log to generate new corpus corresponding to the historical log; using the embedded point data corresponding to the online corpus and the word score table corresponding to the online corpus to determine the eliminated corpus in the online corpus; adding the new corpus to the online corpus and deleting the eliminated corpus from the online corpus to update the online corpus.

[0118] According to the technical solution of the embodiment of the present invention, new corpus corresponding to the historical log is generated, and the consultation texts that users have paid close attention to recently can be screened and repaired to obtain new corpus, which is added to the online corpus, so that the corpus in the online corpus has universality and can continuously follow the hot spots of consultation in the online customer service system; the eliminated corpus is determined by using the buried data and word score table corresponding to the online corpus, and the corpus that needs to be eliminated in the online corpus is automatically selected, so that outdated or low-quality corpus can be deleted; the new corpus is added to the online corpus, and the eliminated corpus is deleted from the online corpus, which can ensure that the number of corpora in the online corpus is maintained within a certain range, reduce operating costs, reduce workload, and avoid the problem of a surge in corpora and the existence of expired corpora that cannot meet user needs due to the lack of a clear elimination mechanism in the prior art.

[0119] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for updating a corpus, characterized in that: include: Filter and compress the historical logs to generate new corpus corresponding to the historical logs; Determine the eliminated corpus in the online corpus by using the buried data corresponding to the online corpus and the word score table corresponding to the online corpus, including: counting the number of times the online corpus segmentation word appears in the online corpus, and determining the word score of the online corpus segmentation word according to the counted number of times, so as to generate the word score table corresponding to the online corpus; judging whether the online corpus meets the preset elimination condition according to the number of exposures corresponding to the online corpus, the number of clicks corresponding to the online corpus and the word score table, and if so, determining the online corpus as the eliminated corpus; The newly added corpus is added to the online corpus, and the eliminated corpus is deleted from the online corpus to update the online corpus.

2. The method according to claim 1, characterized in that The filtering and compressing of the historical logs to generate the new corpus corresponding to the historical logs includes: Obtain historical logs within a first preset time, and decrypt the historical logs to obtain plaintext logs; Filter the plaintext log using preset log filtering rules; Compressing the plaintext log after filtering to obtain candidate corpus corresponding to the plaintext log; Based on a preset screening model, model recognition processing is performed on the candidate corpus to generate the newly added corpus.

3. The method according to claim 2, characterized in that The filtering process of the plain text log by using the preset log filtering rule includes: Perform basic filtering on the plaintext log, filter out system default messages, user related information, URLs, pictures and special characters in the plaintext log, and replace emoticons in the plaintext log; Performing intra-session deduplication processing on the plaintext log to filter out duplicate sessions in each plaintext log; Performing length filtering on the plaintext logs, filtering out the plaintext logs that do not meet the preset character length; The plaintext log is filtered for times, and the logs in the plaintext log that appear less than a preset number of times within a second preset time are filtered out.

4. The method according to claim 3, characterized in that The compressing the filtered plaintext log to obtain candidate corpus corresponding to the plaintext log includes: According to the number of times the plaintext log after the filtering process appears within the second preset time, the plaintext log after the filtering process is compressed to obtain the candidate corpus.

5. The method according to claim 2, characterized in that: The step of performing model recognition processing on the candidate corpus based on the preset screening model to generate the newly added corpus includes: Inputting the candidate corpus into a negative emotion recognition model to identify negative emotion corpus, and inputting the candidate corpus into a sensitive compliance model to identify non-compliant corpus; The negative emotion corpus and the non-compliant corpus are deleted from the candidate corpus to generate the newly added corpus.

6. The method according to claim 1, characterized in that The counting of the number of occurrences of the online corpus segmentation words in the online corpus includes: Acquire at least one online corpus in the online corpus, perform word segmentation on the at least one online corpus, and obtain at least one online corpus word segmentation; For an online corpus segmentation in the at least one online corpus segmentation, count the number of times the online corpus segmentation appears in the online corpus.

7. The method according to claim 6, characterized in that The embedding data includes: exposure data and click data; and Before judging whether the online corpus meets a preset elimination condition according to the number of exposures corresponding to the online corpus, the number of clicks corresponding to the online corpus and the word score table, the method further includes: For an online corpus in the at least one online corpus, the number of exposures corresponding to the online corpus is obtained from the exposure data, and the number of clicks corresponding to the online corpus is obtained from the click data.

8. The method according to claim 1 or 7, characterized in that: The step of judging whether the online corpus meets a preset elimination condition according to the number of exposures corresponding to the online corpus, the number of clicks corresponding to the online corpus, and the word score table, and if so, determining that the online corpus is an eliminated corpus includes: If the exposure times corresponding to the online corpus is zero, the online corpus is determined to be eliminated corpus; If the exposure number corresponding to the online corpus is greater than the first preset exposure number, and the click number corresponding to the online corpus is less than the preset click number, then the online corpus is determined to be eliminated corpus; If the exposure times corresponding to the online corpus are less than the second preset exposure times and greater than zero, the corpus score corresponding to the online corpus is calculated using the word score table. If the corpus score is less than the preset score, the online corpus is determined to be eliminated corpus.

9. The method according to claim 1, characterized in that: After generating the new corpus corresponding to the historical log, the method further includes: Acquire features corresponding to the newly added corpus, and establish a corresponding relationship between the acquired features and the newly added corpus, wherein the features include category type, brand type, and / or consultation entry type; and, Tracking is performed on the newly added corpus to monitor the exposure times and click times of the newly added corpus.

10. A device for updating a corpus, characterized in that: include: A generation module, used to filter and compress historical logs to generate new corpus corresponding to the historical logs; A determination module is used to determine the eliminated corpus in the online corpus by using the embedding data corresponding to the online corpus and the word score table corresponding to the online corpus, including: counting the number of times the online corpus segmentation word appears in the online corpus, and determining the word score of the online corpus segmentation word according to the counted number of times to generate the word score table corresponding to the online corpus; judging whether the online corpus meets the preset elimination condition according to the exposure number corresponding to the online corpus, the click number corresponding to the online corpus and the word score table, and if so, determining the online corpus as the eliminated corpus; An updating module is used to add the newly added corpus to the online corpus and delete the eliminated corpus from the online corpus to update the online corpus.

11. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 9.

12. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images